Yet another blog

Generate your own types in fast-check

2020-04-07T00:00:00.000Z

TLDR

If your custom type is:

1
2
3

class ImaCustom {
    constructor(public s: Set, public a: string[]) {}
}

… and if you’re just getting started with this (like me) and thinking: “ok, now how do I randomly generate my ImaCustom instances?”.

Try generating random sets and arrays which you can then use to create your ImaCustoms.

e.g.

import fc from "fast-check";

test("Some property you want to test", () => {
  fc.assert(
    fc.property(
      // generate [Set, string[]] tuples with some size constraints
      fc.tuple(fc.set(fc.string(100), 25), fc.array(fc.string(100), 50)),
      ([s, a]) => {
        const i = new ImaCustom(s, a);
        // assert your property
      }
    )
  );
});

That’s basically what I wanted to write about 😊

A slow start to a fast-check

I guess you already know what property based testing is if you’re here. Recently, I was testing something… which tbh, I didn’t really know how I wanted to work. But in my mind, I could think of certain properties; certain things which had to hold on the datastructure in question when this operation happens. So - I figured I’d reach for that testing approach I know about; think is amazing; but almost never actually use 🤔

Anyway - it didn’t take long to find fast-check. I had used jsverify before, but I’ve forgotten the API and who cares anyway - I just want to code a few properties and get on with my app… maybe actually get it to a usable state 😅

So… I’m searching online:

Ye, I know more or less what property based testing is. Great - first few examples show how to generate basic types. Cool, found the list of built-in… oh “arbitraries” they’re called… hmm fancy that - and I thought I had an arbitrary name. Now if I could only get to the “how to generate your own flippin’ types” in the documentation and give these stressed eyeballs a break.

Far as I can tell - that section doesn’t exist in the docs. I did eventually have a 🤦‍♂️ moment and realised that any “custom” datastructure must be made up of more basic types.

Show me da code!

This is basically the data I’m working with here (Immutable.js Set / OrderedSet):

export interface ContentProps {
  include?: OrderedSet<string>;
  exclude?: Set<string>;
}

… which is passed in to Content‘s constructor to create an instance. But the constructor has some constraints:

if (include.has("") || exclude.has("")) {
// ... throw Error
}

const invalidValues = include.intersect(exclude);
if (invalidValues.size > 0) {
// ... throw Error
}

So basically, no empty string in the sets and they must be mutually exclusive.

Finally, a Content can sync given a list of file paths string[]… and this is what I came up with for what I had in mind. NOTE: you do not need to understand or even read this code:

public sync(filePaths: string[] = []): Content {
  const filePathsAsSet = Set(filePaths);
  const setOfFilePathsNotAlreadyExcluded = filePathsAsSet.subtract(
    this.exclude
  );
  const oldValidSectionsInRightOrder = this.include.intersect(
    setOfFilePathsNotAlreadyExcluded
  );
  const newValidSections = setOfFilePathsNotAlreadyExcluded.subtract(
    oldValidSectionsInRightOrder
  );

  const include = oldValidSectionsInRightOrder.concat(newValidSections);
  const exclude = filePathsAsSet.intersect(this.exclude);
  return new Content({
    dirName: this.dirName,
    include,
    exclude
  });
}

One property I can think of is this:

In result = content.sync(filePaths: string[]) - any filePath in filePaths which is also in content.exclude should be in result.exclude and not in result.include

This is the property being demoed below, but if you want to get a feel of what sync is about, here are a couple more properties which come to mind:

Any existing values in content.exclude which are not present in filePaths should not be in result.exclude
Any existing values in content.include which are not present in filePaths should not be in result.include
Any existing values in content.include which are in filePaths are kept in the same order in result.include

Generating data

To express this property, I first want to express how the “ingredients” are generated, i.e. the arbitrary data:

const arbitrarySyncTestProps = fc.tuple(
  fc.set(fc.string(100), 25),
  fc.set(fc.string(100), 25),
  fc.array(fc.string(100), 50)
);

So - generate me a tuple of 2 sets of strings with a max size of 25 and whose strings are no more than 100 characters in length. Also, throw in an array of strings for good measure. Super simple - but now I want to get more specific… I should have no empty strings in either Set, nor in the array come to think of it - as those elements are meant to end up in a Set which doesn’t want empty strings. Also, I want the two Sets to be mutually exclusive… and while we’re at it - I think it won’t do to just have random strings. Taking another look at the property:

any filePath in filePaths which is also in content.exclude

So, maybe I should ensure I get some strings which are in both filePaths and exclude:

const arbitrarySyncTestProps = fc
  .tuple(
    fc.set(fc.string(100), 25),
    fc.set(fc.string(100), 25),
    fc.array(fc.string(100), 50)
  )
  .map(([include, exclude, filePaths]) => {
    // make filePathsToAddToExclude so tests can be more meaningful
    let filePathsToAddToExclude = [];
    if (filePaths.length > 0) {
      const numberOfPathsToAddToExclude = Math.floor(
        Math.random() * filePaths.length
      );
      for (let i = 0; i <= numberOfPathsToAddToExclude; i++) {
        filePathsToAddToExclude.push(filePaths[i]);
      }
    }

    // transform the data so it meets Content's pre-conditions
    const includeOrderedSet = OrderedSet(
      include.filter((x) => x !== "" && !exclude.includes(x))
    );
    const excludeSet = Set(
      exclude.concat(filePathsToAddToExclude).filter((x) => x !== "")
    );
    const filePathsArray = filePaths.filter((x) => x !== "");

    return [includeOrderedSet, excludeSet, filePathsArray];
  });

Expressing the property

Finally, the property can be expressed with a few simple assertions:

([include, exclude, filePaths]: [
  OrderedSet<string>,
  Set<string>,
  string[]
]) => {
  const content = new Content({
    include,
    exclude,
  });
  const result = content.sync(filePaths);
  const pathsInBothFilePathsAndExclude: Set<string> = content.exclude.intersect(
    Set(filePaths)
  );

  expect(pathsInBothFilePathsAndExclude.isSubset(content.exclude)).toBe(true);
  expect(result.include.intersect(pathsInBothFilePathsAndExclude).size).toBe(0);
};

Putting it all together

import fc from "fast-check";
import { Set, OrderedSet } from "immutable";
import { Content } from "./Content";

describe("Content", () => {
  describe("content.sync(filePaths: string[])", () => {
    test("Any filePath in filePaths which is also in content.exclude should be in result.exclude and not in result.include", () => {
      const arbitrarySyncTestProps = fc
        .tuple(
          fc.set(fc.string(100), 25),
          fc.set(fc.string(100), 25),
          fc.array(fc.string(100), 50)
        )
        .map(([include, exclude, filePaths]) => {
          // make filePathsToAddToExclude so tests can be more meaningful
          let filePathsToAddToExclude = [];
          if (filePaths.length > 0) {
            const numberOfPathsToAddToExclude = Math.floor(
              Math.random() * filePaths.length
            );
            for (let i = 0; i <= numberOfPathsToAddToExclude; i++) {
              filePathsToAddToExclude.push(filePaths[i]);
            }
          }

          // transform the data so it meets Content's pre-conditions
          const includeOrderedSet = OrderedSet(
            include.filter((x) => x !== "" && !exclude.includes(x))
          );
          const excludeSet = Set(
            exclude.concat(filePathsToAddToExclude).filter((x) => x !== "")
          );
          const filePathsArray = filePaths.filter((x) => x !== "");

          return [includeOrderedSet, excludeSet, filePathsArray];
        });

      fc.assert(
        fc.property(
          arbitrarySyncTestProps,
          ([include, exclude, filePaths]: [
            OrderedSet<string>,
            Set<string>,
            string[]
          ]) => {
            const content = new Content({
              include,
              exclude,
            });
            const result = content.sync(filePaths);
            const pathsInBothFilePathsAndExclude: Set<string> = content.exclude.intersect(
              Set(filePaths)
            );

            expect(
              pathsInBothFilePathsAndExclude.isSubset(content.exclude)
            ).toBe(true);
            expect(
              result.include.intersect(pathsInBothFilePathsAndExclude).size
            ).toBe(0);
          }
        )
      );
    });
  });
});

In conclusion

It was fun to revisit property based testing. One issue I know will come up is the matter of performance. This is taking circa 5 seconds to run so I know its going to be problem. Maybe I’ll only run the property based tests when an env var is set, or configure fast-check to “be more fast!” … somehow (e.g. generate less tests). In any case, it’s still useful and I have already found a couple of issues with my implementation.

Any feedback on this is more than welcome 👇

🍻

Using plain pg with Nest.js

2020-03-26T13:19:25.000Z

Intro

This is going to be a quick demo of how to get started using plain pg (node-postgres) with Nest.js — no ORMs in sight. Having found no simple guide on how to do this when I was looking for it — I figured it’d be worth writing about.

Demo on Github:

https://github.com/justin-calleja/nest-pg-demo

Using plain pg with Nest.js

Create a new Nest.js app — install pg — generate db module

npx nest new nest-pg-demo
cd nest-pg-demo
npm i pg
npx nest g module db

Register a Provider

We’ll use a constants.ts file in /src to be able to import the DI token for our postgres connection (or pool of connections):

1 2	// src/constants.ts export const PG_CONNECTION = 'PG_CONNECTION';

The DI token (dependency injection token) is how you’ll declaratively tell the Inversion of Control (IoC) container which dependency you’d like injected. So you’ll use it to pull in a dependency from the container.

You’ll also use it when you register a provider with the container. Registering a provider is basically associating a DI token (which will be a string in our case) with a way to get a dependency. The container will use this “way to get a dependency” when you request the relevant dependency be injected.

Registering a provider looks like this:

import { Module } from '@nestjs/common';
import { PG_CONNECTION } from '../constants';

const dbProvider = {
    provide: PG_CONNECTION,
    // useValue / useFacotry / useClass: …
};

@Module({
    providers: [dbProvider],
})
export class DbModule {}

i.e. we need an object with the DI token and the way to get the dependency — the dbProvider — and pass it to providers in the Module configuration as shown above.

There’s a couple of ways to specify how to get the dependency in Nest.js. We’ll be using useValue:

import { Pool } from 'pg';
const dbProvider = {
  provide: PG_CONNECTION,
  useValue: new Pool({
    user: 'postgres',
    host: 'localhost',
    database: 'somedb',
    password: 'meh',
    port: 5432,
  }),
};

Finally, we’ll also include the dbProvider in the DbModule’s exports so we’ll be able to use it in other modules importing DbModule:

import { Module } from '@nestjs/common';
import { Pool } from 'pg';
import { PG_CONNECTION } from '../constants';

const dbProvider = {
    provide: PG_CONNECTION,
    useValue: new Pool({
        user: 'postgres',
        host: 'localhost',
        database: 'somedb',
        password: 'meh',
        port: 5432,
    })
};

@Module({
    providers: [dbProvider],
    exports: [dbProvider],
})
export class DbModule {}

Use the Provider

You’ll notice that the AppModule is already importing DbModule (at least if you generated DbModule with the nest cli that should be the case):

// src/app.module.ts
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { DbModule } from './db/db.module';

@Module({
  imports: [DbModule],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

This means that — in our AppModule i.e. in e.g. controllers and providers defined as part of AppModule) we’ll be able to ask Nest’s IoC container to inject a connection for us using the PG_CONNECTION DI token (string).

Doing so in our AppService looks something like this:

// src/app.service.ts
import { Injectable, Inject } from '@nestjs/common';
import { PG_CONNECTION } from './constants';

@Injectable()
export class AppService {
  constructor(@Inject(PG_CONNECTION) private conn: any) {}

  async getUsers() {
    const res = await this.conn.query('SELECT * FROM users');
    return res.rows;
  }
}

Notice that we annotate conn with Inject and make it private in the constructor. This is significant at it allows Nest to know that it should be the one to supply a value for conn and it also knows what to inject via the DI token.

getUsers is just a simple query to use as a sanity check. We’ll use it in AppController:

// src/app.controller.ts
import { Controller, Get } from '@nestjs/common';
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get('users')
  getUsers() {
    return this.appService.getUsers();
  }
}

I guess you might be wondering how we’re able to inject AppService here without the Inject annotation… That’s thanks to a shortcut (or “syntactic sugar” if you prefer). In Nest.js, registering a Provider with a class name:

import { AppService } from './app.service';
@Module({
  providers: [AppService],
})
export class AppModule {}

… is equivalent to:

import { AppService } from './app.service';
@Module({
  providers: [{
    provider: AppService,
    useClass: AppService,
  }],
})
export class AppModule {}

… which means — the DI token doesn’t have to be a string and now Nest.js knows what to inject when we type the private appService in AppController with the type AppService.

Start and seed postgres

Of course, you’re going to want postgres running if you want to try this out. If you have docker installed, take a look at the scripts and sql directories in the repo for this demo — but the gist is:

#!/bin/bash
docker run --rm \
    --name somedb \
    -e POSTGRES_PASSWORD=meh \
    -e POSTGRES_USER=postgres \
    -e POSTGRES_DB=somedb \
    -p 5432:5432 \
    -v somedb-vol:/var/lib/postgresql/data \
    postgres

scripts/start-db.sh starts postgres. --rm will remove the container when we stop it. By virtue of passing in the env vars we do — the container will auto create the somedb database. The volume mapping is so we keep our data even when we stop the container (and have it auto-removed).

After the db is up — you can run scripts/sync-db.sh:

#!/bin/bash
SCRIPTS_DIR=`dirname "$0"`
cat "$SCRIPTS_DIR/../sql/schema.sql" "$SCRIPTS_DIR/../sql/dev-seeds.sql" \
    | psql -U postgres -d somedb -p 5432 -h localhost -1 -f -

This should work regardless of your pwd when you run the script since we’re referencing the sql files relative to the location of the sync-db.sh file on disk (consider this to be an equivalent of __dirname in Node.js). Since schema.sql drops and re-creates the public schema every time — it’s safe to re-run sync-db.sh when you change your table definitions etc…

Start the app and GET /users

Finally, it’s time to run our app with npm run start:dev and hit the /users endpoint with:

1 2	curl localhost:3000/users [{"id":1,"email":"tmp@gmail.com"},{"id":2,"email":"user@gmail.com"},{"id":3,"email":"anotheruser@gmail.com"}]

Trying out Typescript

2016-05-28T00:00:00.000Z

Intro

The intention was to write some demo code to see what problems I come across while just starting to use Typescript, and to document the “work-arounds” along the way. This is a suitable read for anyone who knows what Typescript is, is interested in it, but hasn’t really used it before. You should have Redis installed (as well as Node and Typescript) to follow along.

I’m using the Atom text editor with the atom-typescript package.

A repo for the code in this post is up on Github. Use npm install && typings install to get started.

The demo

We’re going to have 2 scripts to run, pub.js and sub.js. pub.js will just start up a Node REPL exposing a redisClient which we can use to publish messages on channels with. sub.js will be subscribing to channels and listening to messages published by pub.js.

Below is a quick session for illustration:

Simple, but making the compiler happy took a bit of fiddling.

pub.ts

tsconfig.json

$ mkdir redis-ts-eg && cd redis-ts-eg
redis-ts-eg$ npm init --yes
redis-ts-eg$ npm i -S redis
redis-ts-eg$ mkdir src lib

If you now open redis-ts-eg in Atom and create pub.ts (note the ts extension) in src, you should get the following error (and suggestion):

We’re missing a tsconfig.json file and we can get atom-typescript to create one for us using the Atom command-palette by hitting cmd-shift-p (OSX) or ctrl-shift-p (Linux/Windows).

Once you have that open, start typing “create tsconfig.json” and you should get the option to select. This creates the file in src, so just move it one directory up to the project root.

So what is tsconfig.json anyway?

A unified project format for TypeScript… The TypeScript compiler (1.5 and above) only cares about compilerOptions and files
tsconfig.json atom-typescript doc

Ok, so compilerOptions and files are passed to the Typescript compiler, but this isn’t to say that tsconfig.json is an atom-typescript only thing:

Using tsconfig.json:

By invoking tsc with no input files, in which case the compiler searches for the tsconfig.json file starting in the current directory and continuing up the parent directory chain.
By invoking tsc with no input files and a –project (or just -p) command line option that specifies the path of a directory containing a tsconfig.json file.
When input files are specified on the command line, tsconfig.json files are ignored.
tsconfig.json Typescript doc

i.e. both Typescript and atom-typescript use the file (the teams behind each collaborate to avoid conflicts).

In tsconfig.json, filesGlob is a field used by atom-typescript to keep files up to date. i.e. any files matched by filesGlob are automatically (and individually) listed by Atom in files (which is used by tsc, the Typescript compiler).

"filesGlob": [
    "**/*.ts",
    "**/*.tsx",
    "!node_modules/**"
],

typings

Great, so I guess we’re ready to start coding now:

1	import redis = require('redis');

Not so fast. We’re importing redis (Typescript style) but the compiler knows nothing about it. To fix this we’ll use typings, the Typescript Definition Manager, to fetch a description of what the redis module is (i.e. a .d.ts file).

redis-ts-eg$ # Install Typings CLI utility if you don't already have it
redis-ts-eg$ npm install typings --global
redis-ts-eg$ typings --version
1.0.4
redis-ts-eg$ typings search redis
Viewing 5 of 5

NAME            SOURCE HOMEPAGE                                    DESCRIPTION VERSIONS UPDATED
node_redis      dt     https://github.com/mranney/node_redis                   1        2016-03-16T15:55:26.000Z
redis           npm    https://www.npmjs.com/package/redis                     1        2016-05-02T17:09:35.000Z
redis           dt     https://github.com/mranney/node_redis                   1        2016-03-16T15:55:26.000Z
socket.io-redis dt     https://github.com/socketio/socket.io-redis             1        2016-04-01T04:54:12.000Z
ioredis         dt     https://github.com/luin/ioredis                         1        2016-05-21T15:26:53.000Z

You can tell typings to install from one of these sources (assuming a type definition is available at a given source - something you can confirm through searching as done above). npm is the default (configurable through defaultSource in .typingsrc).

1	redis-ts-eg$ typings install redis --save

After installing, you should get a typings.json file and a typings directory. If you save tsconfig.json now, it should update files to:

"files": [
    "src/pub.ts",
    "typings/index.d.ts",
    "typings/modules/redis/index.d.ts"
],

It would work just as well without typings/modules/redis/index.d.ts as typings/index.d.ts references it. In any case, the “Cannot find module ‘redis’” error should be gone now (try closing and re-opening the file if not).

Before continuing, notice that the compiler generates src/pub.js (even when we had an error - Typescript will still transpile to Javascript even with errors). To move the compiler’s output to lib we can add the following to compilerOptions in tsconfig.json:

1
2
3

"compilerOptions": {
  "outDir": "lib",
  ...

Saving pub.ts now should generate pub.js in lib.

We can move on by importing the next module we’ll need and going through a similar process as with the redis module:

1	import repl = require('repl');

repl is a module that comes with the Node distribution, but tsc has no knowledge of it.

redis-ts-eg$ typings search node
Viewing 20 of 212

NAME                          SOURCE HOMEPAGE                                           DESCRIPTION VERSIONS UPDATED
analytics-node                dt     https://segment.com/docs/libraries/node/                       1        2016-03-17T12:06:54.000Z
i18next-node-fs-backend       dt     https://github.com/i18next/i18next-node-fs-backend             1        2016-03-19T10:37:45.000Z
i18n-node                     dt     https://github.com/mashpie/i18n-node                           1        2016-05-21T15:25:45.000Z
jasmine-node                  dt     https://github.com/mhevery/jasmine-node                        1        2016-03-16T15:55:26.000Z
jasmine-node-promise-matchers global                                                                1        2016-03-10T14:35:55.000Z
kafka-node                    dt     https://github.com/SOHU-Co/kafka-node/                         1        2016-03-16T15:55:26.000Z
mocha-node                    dt     http://mochajs.org/                                            1        2016-03-17T12:06:54.000Z
moment-node                   dt     https://github.com/timrwood/moment                             1        2016-05-11T04:33:38.000Z
node                          dt     http://nodejs.org/                                             5        2016-05-23T03:57:54.000Z
node                          env                                                                   5        2016-05-07T21:03:04.000Z
node-4                        dt     http://nodejs.org/                                             1        2016-05-14T16:59:20.000Z
node-array-ext                dt     https://github.com/Beng89/node-array-ext                       1        2016-03-16T15:55:26.000Z
node-azure/azure              dt     https://github.com/WindowsAzure/azure-sdk-for-node             1        2016-03-16T15:55:26.000Z
node-cache                    dt     https://github.com/tcs-de/nodecache                            1        2016-03-17T12:06:54.000Z
node-calendar                 dt     https://www.npmjs.com/package/node-calendar                    1        2016-03-16T15:55:26.000Z
node-config-manager           dt     https://www.npmjs.com/package/node-config-manager              1        2016-03-16T15:55:26.000Z
node-dir                      dt     https://github.com/fshost/node-dir                             1        2016-03-16T15:55:26.000Z
node-ffi                      dt     https://github.com/rbranson/node-ffi                           1        2016-03-16T15:55:26.000Z
node-ffi/node-ffi-buffer      dt                                                                    1        2014-06-18T21:31:11.000Z
node-fibers                   dt     https://github.com/laverdet/node-fibers                        1        2016-03-17T12:06:54.000Z

After searching, you’ll see we get two options from the list above, one with a source of dt, and one with a source of env. I’ve tried installing both and they’re identical in their repl definitions so we’ll just pick one:

1	redis-ts-eg$ typings install dt~node --save --global

From what I can tell, we need to use --global with this one because of the contents of the definitions in its .d.ts (i.e. typings/globals/node/index.d.ts). Unlike the redis one, not everything in this .d.ts file is wrapped around a module declaration which exports the desired definitions. Instead, there are global definitions like these:

declare var process: NodeJS.Process;
declare var global: NodeJS.Global;

declare var __filename: string;
declare var __dirname: string;

declare function setTimeout(callback: (...args: any[]) => void, ms: number, ...args: any[]): NodeJS.Timer;
declare function clearTimeout(timeoutId: NodeJS.Timer): void;
// etc…

This is as opposed to contained (wrapped in a module) type definitions like the ones for redis which are scoped to a variable after importing the redis module.

Apparently, typings can make this distinction because it will fail to install dt~node without the --global flag.

In any case, the compiler now knows about repl so we can finish off pub.ts:

import redis = require('redis');
import repl = require('repl');

let redisClient = redis.createClient();

let replServer = repl.start({
  prompt: '>> '
});

replServer.context.redisClient = redisClient;

Which gives us our next challenge:

Adding context

The following is what the compiler knows about the repl module from typings/globals/node/index.d.ts (which should be automatically added to your tsconfig.json thanks to atom-typescript):

declare module "repl" {
    import * as stream from "stream";
    import * as events from "events";

    export interface ReplOptions {
        prompt?: string;
        input?: NodeJS.ReadableStream;
        output?: NodeJS.WritableStream;
        terminal?: boolean;
        eval?: Function;
        useColors?: boolean;
        useGlobal?: boolean;
        ignoreUndefined?: boolean;
        writer?: Function;
    }
    export function start(options: ReplOptions): events.EventEmitter;
}

No mention of context anywhere. The easiest thing to do would be to add it right there. But what are we going to do in the next project which needs the same thing? Repeat the operation? Not fun.

At this point you could start managing your own index.d.ts for Node. Maybe submit a pull request with your changes and hope they get added. By managing your own type definitions in a repo you manage, you can import them into your code and re-use them.

However, for a throw-away project like this, I’d rather not have to stay doing this. Instead, I’ll just add a typingslocal directory to this project and add type definitions extending existing ones there.

Note: the name is important if you want to keep atom-typesript‘s auto management of the files field in tsconfig.json. We want definition files under typingslocal to be listed below those in typings. That should be the case with typingslocal (it is for me), but if for some reason this ordering changes with whatever version of atom-typescript you’re using, it’s best to set rewriteTsconfig to off (in tsconfig.json) and manage files yourself. That way you can make sure that d.ts files in typingslocal are listed after the ones in typings in tsconfig.json‘s files field.

It seems to me that all we need to do in this case is define our own type extending EventEmitter, giving it a context and using that type for the return value of start:

// typingslocal/global/node/index.d.ts
declare module "repl" {
  import * as events from "events";

  interface ReplEventEmitter extends events.EventEmitter {
    context: any;
  }

  export function start(options: ReplOptions): ReplEventEmitter;
}

After saving tsconfig.json, files should refresh and the error in pub.ts should go away.

sub.ts

Moving on to sub.ts - if you copy-paste the following, you’ll find that the compiler is only unaware of the startsWith method on strings:

import redis = require('redis');

let redisClient = redis.createClient();

let defaultChannel = 'helloChannel';
let args = process.argv.slice(2);
let channelName: string = args[0] || defaultChannel;

function subscribe(redisClient, channelName) {
  redisClient.subscribe(channelName, (err, channel) => {
    if (err) throw err;
    console.log(`now listening on channel: '${channelName}'`);
  });
}

redisClient.on('error', console.error);

redisClient.on('message', (channelName, msg: String) => {
  console.log(`${channelName}: ${msg}`);
  if (msg === '/unsubscribe') {
    redisClient.unsubscribe(channelName);
  } else if (msg.startsWith('/subscribe')) {
    let channel = msg.split(' ').slice(1).join(' ');
    subscribe(redisClient, channel);
  }
});

redisClient.on('unsubscribe', (channelName, count) => {
  console.log(`unsubscribing from ${channelName}. Client is now listening to ${count} channel(s)`);
  if (count === 0) {
    console.log('no more subscriptions; exiting…');
    process.exit();
  }
});

redisClient.on('subscribe', (channelName, count) => {
  console.log(`listening on channel: ${channelName}. Client is now listening to ${count} channel(s)`);
});

subscribe(redisClient, channelName);

Clearly, we need to teach the compiler about startsWith (Node, which is what we’ll be running on, has this method already defined on the String prototype… we just need to make the compiler happy).

I came across a way to do this here. It is based on “global augmentation”:

TypeScript also has the notion of global augmentations of the form declare global { }. This allows modules to augment global types such as Array if necessary.
Augmenting global/module scope from modules

An example of augmenting Array is given in the 1.8 release notes:

// Ensure this is treated as a module.
export {};

declare global {
    interface Array {
        mapToNumbers(): number[];
    }
}

Following this pattern, we can fix our issue with:

// typingslocal/globals/runtime/index.d.ts
export {}

declare global {
  interface String {
    startsWith(searchString: string, position?: number): boolean;
  }
}

Note: if the use of an empty export to ensure the file is treated as a module doesn’t make sense to you, read this.

You should now have no compiler errors and be able to run the 2 scripts as shown in the demo at the start of this post.

Conclusion

This post was meant as a “how-to” for people (and by someone) just getting started with Typescript. We’ve seen a lot of friction from the compiler and zero opportunities for it to help us (using JS would have been much easier for this demo). This can be expected to change as we start defining our own types and our code base gets larger :)

Serving a Webpack bundle in Spring Boot

2016-04-17T00:00:00.000Z

Intro

This post demos an example of using Webpack to bundle a frontend project and Maven to package it up in a jar and include it in a Spring Boot app to serve it.

The example in its final state is up on Github.

Getting started

We’ll start by creating the 3 following projects:

parent
fe
be

parent is just a convenient way to package both our other 2 modules up in one go. It only consists of the following pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0modelVersion>
  <groupId>com.tmpgroupId>
  <artifactId>parentartifactId>
  <version>0.0.1-SNAPSHOTversion>
  <packaging>pompackaging>

  <modules>
    <module>../femodule>
    <module>../bemodule>
  modules>
project>

That way we can mvn clean install and it will install both our other 2 modules.

The frontend

fe will be our frontend module. It’ll mostly feature JavaScript technology, but since we’re aiming to include this in a Spring Boot app, we’ll need to package it up in a jar file. We’ll be using the frontend-maven-plugin to do this as shown in fe/pom.xml below:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0modelVersion>

  <groupId>com.tmpgroupId>
  <artifactId>feartifactId>
  <version>0.0.1-SNAPSHOTversion>
  <packaging>jarpackaging>

  <build>
    <plugins>
      <plugin>
        <groupId>com.github.eirslettgroupId>
        <artifactId>frontend-maven-pluginartifactId>
        <version>0.0.27version>

        <executions>

          <execution>
            <id>install node and npmid>
            <goals>
              <goal>install-node-and-npmgoal>
            goals>
            <configuration>
              <nodeVersion>v4.2.1nodeVersion>
              <npmVersion>3.5.3npmVersion>
            configuration>
          execution>

          <execution>
            <id>npm installid>
            <goals>
              <goal>npmgoal>
            goals>
            <configuration>
              <arguments>installarguments>
            configuration>
          execution>

          <execution>
            <id>npm run buildid>
            <goals>
              <goal>npmgoal>
            goals>
            <configuration>
              <arguments>run buildarguments>
            configuration>
          execution>

        executions>
      plugin>
    plugins>
  build>
project>

The gist is that this plugin will install Node and npm as well as our npm dependencies, and build the project via npm run build. As you can see, we’re specifying the Node and npm versions to install. These exact versions are downloaded if not found in the project in question irrespective of which versions of this software may (or may not) already exist on the machine you’re building on (making for repeatable builds on different machines).

Next, we’ll define our build npm script in our package.json:

{
  "name": "fe",
  "version": "0.0.1-SNAPSHOT",
  "description": "",
  "main": "./app",
  "scripts": {
    "build": "webpack -p"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "webpack": "1.12.9",
    "webpack-dev-server": "1.14.0"
  }
}

We’re simply running webpack -p to build for production (i.e. the bundle will be minified). Once you install the npm dependencies listed here (either through the Maven plugin or by running npm install), you’ll get a local copy of webpack in your node_modules directory. In node_modules, you’ll also get a .bin directory housing any scripts specified as binaries in any installed node modules.

So for instance, node_modules/webpack/package.json has the following file specified as a binary:

1
2
3

"bin": {
  "webpack": "./bin/webpack.js"
},

This means there will be a link to the file node_modules/webpack/bin/webpack.js in node_modules/.bin/, and this link will have the name webpack (the key used in the bin object above).

Additionally, anything in node_modules/.bin will be on your PATH when you run the npm scripts specified in your package.json. i.e. webpack will be on your PATH when you run npm scripts through npm such that you don’t need to do the following:

1
2
3

"scripts": {
  "build": "./node_modules/.bin/webpack -p"
},

You can simply use webpack. npm effectively alters your environment before running the scripts you tell it to run.

But Webpack needs to be configured to do anything for us - so we’ll do that next (fe/webpack.config.js):

var packageJSON = require('./package.json');
var path = require('path');
var webpack = require('webpack');

const PATHS = {
  build: path.join(__dirname, 'target', 'classes', 'META-INF', 'resources', 'webjars', packageJSON.name, packageJSON.version)
};

module.exports = {
  entry: './app/index.js',

  output: {
    path: PATHS.build,
    filename: 'app-bundle.js'
  }
};

We’re specifying ./app/index.js as our entry point. This script, along with anything else it pulls in by requireing it, will be bundled up in a file named app-bundle.js in the directory: __dirname + '/target/classes/META-INF/resources/webjars/fe/0.0.1-SNAPSHOT'.

This might seem like a peculiar output destination. By choosing target/classes, Maven will include Webpack’s output in the final product for this module, the jar file. We could have chosen something like src/main/resources which would have had a similar effect, but then running mvn clean would not delete Webpack’s output (something we want to do when cleaning).

In passing, note the /META-INF/resources part of the path we’re using. Containers implementing the Servlet 3.0 specification should be able to serve assets in a jar’s /META-INF/resources without any configuration. However, we’ll be explicitly serving our Webpack build via Spring configuration.

Next, let’s write some simple code just to have something to bundle up:

fe/app/index.js:

var greeter = require('./greeter');

var greeting = greeter.greet();

if (typeof document !== 'undefined') {
  var el = document.createElement('h1');
  el.innerHTML = greeting;
  document.body.appendChild(el);
} else {
  console.log(greeting);
}

fe/app/greeter.js:

module.exports = {
  greet: function(name) {
    return name ? 'Hello ' + name + '!' : 'Hello World!';
  }
}

Notice how in fe/app/index.js we’re guarding against not having a document object. This isn’t really necessary as we intend to write a frontend app, but it’s interesting to note how we can get this code to work outside the browser. It is still an npm package and we could publish it to a private or public registry and re-use it elsewhere - maybe in an app that doesn’t use this module as a frontend.

Running mvn clean install should now give you the jar we’ll be using from our backend.

The backend

Our backend’s pom.xml:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0modelVersion>

  <groupId>com.tmpgroupId>
  <artifactId>beartifactId>
  <version>0.0.1-SNAPSHOTversion>

  <properties>
    <java.version>1.8java.version>
  properties>

  <parent>
    <groupId>org.springframework.bootgroupId>
    <artifactId>spring-boot-starter-parentartifactId>
    <version>1.3.3.RELEASEversion>
  parent>

  <dependencies>
    <dependency>
      <groupId>org.springframework.bootgroupId>
      <artifactId>spring-boot-starter-webartifactId>
    dependency>
    <dependency>
      <groupId>org.springframework.bootgroupId>
      <artifactId>spring-boot-devtoolsartifactId>
      <optional>trueoptional>
    dependency>
    <dependency>
      <groupId>com.tmpgroupId>
      <artifactId>feartifactId>
      <version>0.0.1-SNAPSHOTversion>
    dependency>
  dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.bootgroupId>
        <artifactId>spring-boot-maven-pluginartifactId>
      plugin>
    plugins>
  build>
project>

With the pom in place, we can start our configuration in be/src/main/java/com/tmp/App.java:

package com.tmp;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationListener;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;

@EnableWebMvc
@SpringBootApplication
public class App implements ApplicationListener<ContextRefreshedEvent> {

    private static final Logger LOG = LoggerFactory.getLogger(App.class);

    @Value("${spring.profiles.active}")
    protected String springProfilesActive;

    @Override
    public void onApplicationEvent(ContextRefreshedEvent event) {
        LOG.info("=======================================");
        LOG.info("App running with active profiles: {}", springProfilesActive);
        LOG.info("=======================================");
    }

    public static void main(String[] args) throws Exception {
        SpringApplication.run(App.class, args);
    }

}

It’s mostly annotation driven:

@SpringBootApplication = @Configuration + @EnableAutoConfiguration + @ComponentScan.

@EnableWebMvc = Adds the configuration in WebMvcConfigurationSupport which you can then override as necessary.

I’m also adding some superfluous things like implementing ApplicationListener to log out the active profile when the ApplicationContext gets initialized or is refreshed (i.e. to log on server startup). We don’t actually need this, but I’ll be going off on a bit of a tangent when using the webpack-dev-server later in this post. We’ll then need to enable CORS from our Spring Boot server. Hence, I’m logging out the active profile on start up to show the environment the server is running in (as CORS will only be enabled if the server is running in dev mode). Plus I just like logging things like this on startup.

For more on logging info on app startup you might be interested in this (what originally got me to start doing this in Spring apps).

Our configuration continues in be/src/main/java/com/tmp/WebMvcConfig.java which is responsible for configuring beans to do with the web side of things:

package com.tmp;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.web.servlet.config.annotation.CorsRegistry;
import org.springframework.web.servlet.config.annotation.ResourceHandlerRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter;

@Configuration
public class WebMvcConfig extends WebMvcConfigurerAdapter {

    private static final String[] CLASSPATH_RESOURCE_LOCATIONS = { "classpath:/public/" };

    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
        super.addResourceHandlers(registry);
        if (!registry.hasMappingForPattern("/webjars/**")) {
            registry.addResourceHandler("/webjars/**").addResourceLocations("classpath:/META-INF/resources/webjars/");
        }
        if (!registry.hasMappingForPattern("/**")) {
            registry.addResourceHandler("/**").addResourceLocations(CLASSPATH_RESOURCE_LOCATIONS);
        }
    }

}

We’re adding 2 static asset resource handlers. Our server will look in classpath:/META-INF/resources/webjars/ for any requests starting with /webjars/. So when our fe project is all packaged up and on the classpath, since it contains the following file: META-INF/resources/webjars/fe/0.0.1-SNAPSHOT/app-bundle.js, a request such as http://localhost:8090/webjars/fe/0.0.1-SNAPSHOT/app-bundle.js should serve that file as expected.

But doing so will only get you a bunch of JavaScript source code displayed in your browser - which is why I’ve added an “anything else” pattern (i.e. /**) and mapped it to search in classpath:/public (another example of this here).

Now, since the contents of src/main/resources in Mavenized projects end up packaged in your jar file, we can put our static assets (like index.html below) in be/src/main/resource/public/ and they will be served from the classpath by our server:


<html>
<head>
<meta charset="UTF-8">
<title>Apptitle>
head>
<body>

<script src="/webjars/fe/0.0.1-SNAPSHOT/app-bundle.js">script>
body>
html>

One final thing, add application.properties to src/main/resources:

1 2	spring.profiles.active=dev server.port=8090

This gets picked up auto-magically via Spring Boot configuration.

At this point you should be able to start up the server (either through your IDE or by running mvn clean package && java -jar ./target/be-0.0.1-SNAPSHOT.jar in be), hit http://localhost:8090/index.html and get “Hello World!” back.

You will notice that leaving off the index.html part of the URL will break things. We can quickly fix this by following the advice in this SO answer by Dave Syer.

i.e. we can add the following to WebMvcConfig (note: if you’re using the spring-boot-devtools and it’s still not working after adding this change, try manually re-starting the server):

@Override
public void addViewControllers(ViewControllerRegistry registry) {
    registry.addViewController("/").setViewName("forward:/index.html");
}

webpack-dev-server

Fast feedback is cool and healthy. We currently have a process to bundle our fe app into a jar and serve these assets via our Boot app, which is great when we’re all done developing our frontend code. However, it’s not something you want to do while actually building your frontend code as it’s too time consuming and soul crushing.

You might consider writing a simple Express app and serve your Webpack generated bundles from it while developing. At least you’d skip the Maven build. That would be a start. You’d also need to watch files and build them with Webpack unless you fancy manually running Webpack when you’re done changing files and possibly re-starting the Node server (unless this dev server is not caching the built files).

Good thing this Express based server is already written for us. It’s called webpack-dev-server and we’ll use it to serve Webpack’s output as we make changes to our code. (Note: in case you already have an Express app and you want to bake in Webpack’s file watching and bundling, you can use the webpack-dev-middleware).

To start the dev server you first need to install it. As we’ve listed it as a devDependency in our package.json above, you could run it with ./node_modules/.bin/webpack-dev-server, or you could install it globally with npm i -g webpack-dev-server (after which you can just run webpack-dev-server to run it).

Of course, before running the server, our frontend code needs to run on a web page so we’ll need some kind of HTML file to require the bundle we’re building through Webpack. To do this, we could use the html-webpack-plugin, but for the sake of transparency, we’ll add the following in fe/tmp/index.html:


<html>
<head>
<meta charset="UTF-8">
<title>Apptitle>
head>
<body>

<script src="../assets/app-bundle.js">script>
body>
html>

Now you might be wondering where assets came from? It’s the output.publicPath we’re still missing in our webpack.config.js, and which you can add like so:

// ...
  output: {
    path: PATHS.build,
    publicPath: '/assets/',
    filename: 'app-bundle.js'
  }

From the docs, publicPath is the path from which the webpack-dev-server will serve the bundles created by Webpack:

The Webpack Dev Server also uses this to determine the path where the output files are expected to be served from
output.publicPath doc

(Note: I used /assets/ to keep with the example in the docs. In our case, since our HTML is in the tmp directory, it would probably make more sense to set the publicPath to /tmp/. The src in our script tag would then be a simpler: app-bundle.js).

We’re finally ready to see this in action. Assuming you’re in fe, run the server with the –inline and –hot arguments: webpack-dev-server --inline --hot (you might want to add this as an npm script as we’ve done with build above).

You should now be able to hit http://localhost:8080/tmp/index.html and get:

http://localhost:8080/assets/app-bundle.js is the request our HTML file makes to get the bundle built by Webpack. Note that this is content which is served from memory i.e. when using webpack-dev-server bundles like app-bundle.js are not saved to disk. They are (re)generated on file changes (the files which make up our bundles). If you want to hit the disk, you can run Webpack itself - not the dev server - as we have through npm above.

Now, notice the console output in the screen shot above. If you have that, then hot module replacement is in place and you can go ahead and change fe/app/index.js or fe/app/greeter.js and see the changes in your browser after saving and without re-loading the page… WIN!

CORS in development

Things are looking good. There is, however, one issue we’ll have to face as soon as our app graduates from the “hello world” level. Since our frontend assets are being served from a different server than the one serving our backend data (on a different port), we’re going to have to enable CORS requests originating from localhost:8080 (our webpack-dev-server).

Doing so is simple enough in Spring Boot. Just add the following to be/src/main/java/com/tmp/WebMvcConfig.java:

@Bean
@Profile("dev")
public WebMvcConfigurer corsConfigurer() {
    return new WebMvcConfigurerAdapter() {
        @Override
        public void addCorsMappings(CorsRegistry registry) {
            registry.addMapping("/api/**").allowedOrigins("http://localhost:8080");
        }
    };
}

So, when our server is running in dev mode, any requests to /api/** coming from http://localhost:8080 will be allowed.

Next, if we want to try this out, we’re going to need at least one REST endpoint to work with:

be/src/main/java/com/tmp/GreeterController.java:

package com.tmp;

import java.time.LocalDateTime;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;

@Controller
@RequestMapping(value = "/api/greetings")
public class GreeterController {
    
    private static final String template = "Hello, %s!";

    @RequestMapping(method = RequestMethod.GET, produces = { "application/json" })
    public @ResponseBody Greeting greetNoName() {
        return new Greeting(String.format(template, "World"));
    }

    @RequestMapping(value = "/{name}", method = RequestMethod.GET, produces = { "application/json" })
    public @ResponseBody Greeting greetName(final @PathVariable String name) {
        return new Greeting(String.format(template, name));
    }

    public class Greeting {
        private final String content;
        private final LocalDateTime time;

        public Greeting(String content) {
            this.content = content;
            this.time = LocalDateTime.now();
        }

        public String getContent() {
            return this.content;
        }

        public String getTime() {
            return this.time.toString();
        }
    }

}

This is straightforward enough. We can hit /api/greetings/{name} with an optional {name} path parameter and get back some JSON.

Finally, we can change our client (fe/app/index.js) to something like the following:

var greeter = require('./greeter');

var greeting = greeter.greet();

if (typeof document !== 'undefined') {
  var apiEndpoint = 'http://localhost:8090/api/greetings';
  var el = document.createElement('h1');

  fetch(apiEndpoint + '/webpack').then(function(response) { 
    return response.json();
  }).then(function(obj) {
    el.innerHTML = greeting + '
' + obj.content + '
At ' + obj.time;
    document.body.appendChild(el);
  }).catch(function(err) {
    el.innerHTML = 'oh no…';
    document.body.appendChild(el);
  });
} else {
  console.log(greeting);
}

You can try commenting out and undoing the CORS configuration in our Spring app to see it’s effect in the browser as shown below (Note: you don’t need to restart the Spring Boot server either thanks to the spring-boot-devtools plugin):

Finally, to wrap things up, we can cd into our parent project and run mvn clean install to package and install our fe and be. You can then run the be jar with:

java -jar ./$PATH_TO_JAR/be-0.0.1-SNAPSHOT.jar --spring.profiles.active=prod

where $PATH_TO_JAR is either be/target or its location in your local maven repo.

Hitting http://localhost:8090/index.html should give you the same output as when running with both servers.

Elixir Streams reload

2016-01-26T00:00:00.000Z

Intro

This post is based on Drew Olson‘s Elixir Streams. Since I’m just getting started with Elixir, I came across a few issues understanding the “Building an API with Streams” section in Drew’s post. This post is my attempt at breaking it down while learning about Elixir.

I’m using Elixir 1.2.0 (and Erlang 18.1).

Intro some more

The code for the main example is up on Github
There is a How to install Elixir with asdf section at the end if you don’t have Elixir and want a quick way of installing it
I will not be using the exact Github endpoint (e.g. https://github.com/api/v3/orgs/elixir-lang/repos) used in Drew’s post because:
- I’m not familiar with the Github API
- Couldn’t get the given endpoint to work after generating an access token
- Got the data I wanted anyway using the endpoint in this post (no need for access tokens)
- It cuts down on code :)
This post is primarily a “learning devise”, by which I mean I intend to side track from the main intention below to explain (or mention and link to) Elixir basics. i.e. this post is not really about Elixir Streams although it features them. The original idea for this post was to learn some Elixir basics in the context of the app in Drew’s “Building an API with Streams” section.
Code example in the “Building an API with Streams” section in Drew’s post has been tweaked to work with Elixir version 1.2.0.
This is a long and winding one… - you have been warned :)

The intention

We want to write an Elixir module to expose an Elixir based API around the Github REST API in such a way as to hide pagination from users of our API. Our API will give users streams to work with. As we’ll see, this means we never hit Github unless the user actually reads from the stream, and we only paginate as much as necessary (depending on how much the user reads from the stream and how many pages Github has for the particular resource).

It’s probably best to go over this intention in more detail.

How to fetch Github organization repos and their pagination

We will only be concerned with a single endpoint from Github’s API in this post, namely, listing the repositories of a specific organization. So let’s start off with the elixir-lang organization. If you open the following in a browser: https://api.github.com/orgs/elixir-lang/repos, you should get back something like:

[
  {
    "id": 1234714,
      "name": "elixir",
      "full_name": "elixir-lang/elixir",
      ...
  },
  {
    "id": 1467845,
    "name": "elixir-tmbundle",
    "full_name": "elixir-lang/elixir-tmbundle",
    ...
  },
  ...
]

Opening your browser’s dev tools, you’ll also be able to look at the HTTP response headers from the API (or curl -i https://api.github.com/orgs/elixir-lang/repos | less if you’re on a Unixy system). At the time of writing, you will see that there is no Link header for elixir-lang as it hasn’t got enough repos to necessitate pagination.

Maybe there’s a parameter to limit the number of returned repos - in which case, you’d be able to trigger pagination like that. But I’m too lazy to look into that so I’ll just pick something like: https://api.github.com/orgs/nodejs/repos, which gives back this Link header:

1 2	Link: <https://api.github.com/organizations/9950313/repos?page=2>; rel="next", <https://api.github.com/organizations/9950313/repos?page=3>; rel="last"

Using a tool like jq, we can easily count how many repos we just got back:

1 2	$ curl -s "https://api.github.com/orgs/nodejs/repos" \| jq '. \| length' 30

So now we know that we’ll only ever get a maximum of 30 repos for each such request we make to Github. If the organization happens to have more than 30 repos, we’ll need to follow the link with rel="next", and so on, until there are no more repos to get - i.e. we need to paginate.

But, we don’t want to paginate unless the user actually wants more data - which is why we give the user a stream.

Quick intro to streams

The thing about streams is… they’re lazy… and they’re enumerable.

Stream the Elixir module, like the Enum module, works on Enumerable things. But a stream itself is some kind of data structure which implements the Enumerable protocol (making it an Enumerable thing). This means that streams, like other Enumerables, implement:

Enumerable.count/1
Enumerable.member?/2
Enumerable.reduce/3

just like all the following Enumerables (checkout elixir):

~/github-stuff/elixir$ egrep -R -n --exclude-dir={bin,\.git} ".*defimpl Enumerable.*" .
./lib/elixir/lib/enum.ex:2748:defimpl Enumerable, for: List do
./lib/elixir/lib/enum.ex:2760:defimpl Enumerable, for: Map do
./lib/elixir/lib/enum.ex:2783:defimpl Enumerable, for: Function do
./lib/elixir/lib/file/stream.ex:68:  defimpl Enumerable do
./lib/elixir/lib/gen_event/stream.ex:58:defimpl Enumerable, for: GenEvent.Stream do
./lib/elixir/lib/hash_dict.ex:220:defimpl Enumerable, for: HashDict do
./lib/elixir/lib/hash_set.ex:235:defimpl Enumerable, for: HashSet do
./lib/elixir/lib/io/stream.ex:49:  defimpl Enumerable do
./lib/elixir/lib/map_set.ex:266:  defimpl Enumerable do
./lib/elixir/lib/range.ex:66:defimpl Enumerable, for: Range do
./lib/elixir/lib/stream.ex:1222:defimpl Enumerable, for: Stream do

That’s great, but maybe not so practical right now. Let’s focus on the lazy property of streams which is what we’ll leverage to paginate on an “as-needed” basis.

Being lazy, you can define any transformation you want on streams using the API in the Stream module, but doing so will not trigger the stream’s enumeration. This means that “nothing” will actually happen when you define a stream or make one from another using the Stream module’s API.

Consider the following (adapted from the online doc):

iex(1)> stream = Stream.map([1, 2, 3], fn(x) -> IO.puts "x is #{x}"; x * 2 end)
#Stream<[enum: [1, 2, 3], funs: [#Function<44.120526864/1 in Stream.map/2>]]>
iex(2)> Enum.to_list(stream)
x is 1
x is 2
x is 3
[2, 4, 6]

In our first expression (iex(1)>), we are creating a stream from a list and a function, using Stream.map/2. The REPL shows us that we got back a stream from Stream.map/2. It shows us a human readable string based on the Inspect protocol. As a side note:

Keep in mind that, by convention, whenever the inspected value starts with #, it is representing a data structure in non-valid Elixir syntax.
Protocols

In other words, the stream variable is now bound to a stream - “some kind of data structure” - the innards of which you should make no assumptions on (as they might change in different versions of Elixir).

In our second expression (iex(2)>), we’re using Enum.to_list/1 to convert the stream data structure to an Erlang/Elixir list. On doing so, to_list/1 is internally calling Enumerable.reduce/3 on the stream we pass to it which will “force” the lazy stream to produce something. In the case of to_list/1, the stream is “forced” to keep producing values until it’s exhausted.

That is why we don’t see the side effect of printing to stdout until we actually trigger the stream’s enumeration.

Of course, we don’t have to exhaust the stream. We can just take what we want from it, e.g. using Enum.take/2:

1
2
3

iex(3)> Enum.take(stream, 1)
x is 1
[2]

But consider this:

1 2	iex(4)> Enum.take([4, 5, 6], 1) [4]

What’s different here? The difference is that the data you take/2 from stream doesn’t actually exist until you take it, whereas that from [4, 5, 6] exists in memory before ever calling take/2 on it. Granted, in this case, stream is lazily producing data from a list data source which exists in memory (i.e. [1, 2, 3]). This might make the distinction a little more nuanced for this particular example, but what’s important to keep in mind is that the actual values the stream produces (1, 2, and 3) don’t exist in memory until the stream is actually consumed. Our stream could just as easily be feeding our program data from a database or a REST API instead of an in-memory data structure.

For example, it could be giving Enum.take/2 data it got from its first HTTP request to Github’s REST API. If take/2 doesn’t want more than 30 repos, then stream only ever needs to make 1 HTTP request. If take/2 wants more, stream would have to paginate as necessary but at least take/2 doesn’t have to wait until stream makes enough requests to exhaust the number of repos for a particular organization before it starts receiving data from stream.

That is exactly what we want for the Elixir API we’ll be writing.

But first…

Know your poison

The example coming up next makes use of HTTPoison as well as the more mainstream (but no less deadly) Poison modules - so it makes sense to cover basic usage of each before digging in.

Bootstrap a new project with mix new github; cd into it, and add these modules as dependencies in mix.exs:

1
2
3

defp deps do
  [{:httpoison, "~> 0.8.0"}, {:poison, "~> 1.5"}]
end

Then run mix deps.get to fetch the poisonous bunch.

HTTPoison

First things first. Unless you fancy manually starting the HTTPoison applicaiton every time you load iex (with HTTPoison.start), add it as an application in mix.exs:

1
2
3

def application do
  [applications: [:logger, :httpoison]]
end

It seems that some processes need to be running before using HTTPoison and adding :httpoison in the applications keyword list will automatically make this happen. I’m guessing that modules listed in applications have their start function called by mix which then spawn the processes, but I’m not entirely sure how that works.

In HTTPoison’s case, Erlang’s application:ensure_all_started/1 seems to be what’s making this happen.

In any case, starting iex and loading our app with iex -S mix (which starts a shell with the project and all of its dependencies loaded and started), we can now start experimenting a bit:

1 2	iex(1)> HTTPoison.get! "https://api.github.com/orgs/elixir-lang/repos" # ...

should give you back a bunch of data with a status code of 200. From the example request in the project’s README.md (and from this), we can see that the response we get back is a HTTPoison.Response struct with an integer status_code, a binary body, and a list for headers:

1 2	iex(2)> %HTTPoison.Response{body: body, headers: headers, status_code: status_code} = v(1) # ...

Above, we’re getting the evaluation of expression 1 in iex (i.e. the result of iex(1)>) and extracting its individual parts using pattern matching (we’re using the v(n \\ -1) iex helper function to do this - and note that the double backslash syntax in the helper function’s doc is for default arguments). We now have the following bound variables in our iex session: body, headers, and status_code.

Next, add lib/github_gateway.ex:

# lib/github_gateway.ex
defmodule Github.Gateway do
  use HTTPoison.Base

  @endpoint "https://api.github.com"

  def endpoint do
    @endpoint
  end

  defp process_url(url) do
    @endpoint <> url
  end
end

There’s a couple of things to note here. The easiest to explain is the definition of the @endpoint module attribute which serves as a constant. At compile time, usage of this attribute is changed to the Github endpoint we’ve set it to.

We’re also using use HTTPoison.Base, and as we can see from the online doc, this is compiled to something like:

defmodule Github.Gateway do
  require HTTPoison.Base
  HTTPoison.Base.__using__ []
  # ...
end

Behind the scenes, use requires the given module and then calls the __using__/1 callback on it allowing the module to inject some code into the current context.
Getting started guide

(note: I’m passing __using__/1 an empty list as an argument above. This list is an empty keyword list, which means that if it weren’t empty (i.e. if we had passed any options to the module we were useing), they would be present in this list when the use macro is expanded in the form of 2-element tuples with an atom for the first element).

requireing seems to be necessary to “guarantee” that HTTPoison.Base is available during compilation:

Macros are chunks of code that are executed and expanded at compilation time. This means, in order to use a macro, we need to guarantee its module and implementation are available during compilation. This is done with the require directive
Getting started guide

However, when the compiler is processing this, “code injection” (or “macro expansion”) isn’t over after just the first step above (the expansion of the use macro) because HTTPoison.Base.__using__/1 is itself a macro, so it needs to be expanded too.

If you have just a quick look at HTTPoison.Base.__using__/1, you’ll see that it’s defining a bunch of functions within a quote block. Basically, we can write normal Elixir code within quote blocks and all that code will be transformed into a data structure which is understood by, and fed to, the Elixir compiler during macro expansion.

Effectively, it’s as if the functions in this quote block were defined in our module, Github.Gateway.

You can confirm this by removing the use HTTPoison.Base expression in Github.Gateway (or replacing use with require without calling the __using__/1 macro). If you then iex -S mix and hit the key after Github.Gateway., it will expand to the only thing available in that module at this point, the public endpoint function. If you use HTTPoison.Base and try the same thing, you’ll get a list of the injected code:

iex(1)> Github.Gateway.
delete!/3     delete/3      endpoint/0    get!/3        get/3
head!/3       head/3        options!/3    options/3     patch!/4
patch/4       post!/4       post/4        put!/4        put/4
request!/5    request/5     start/0

As you can see, we now have get!/3, the same function we used from HTTPoison just a minute ago. In fact:

Under the hood, the HTTPoison module just uses HTTPoison.Base… without overriding any default function.
HTTPoison docs

However, since we’ve defined our own process_url/1 function, we don’t need to specify the full URL to make a similar request to the one we’ve made before:

1 2	iex(1)> Github.Gateway.get! "/orgs/elixir-lang/repos" # ...

In fact, we can’t specify the full URL as a parameter since we’ve changed the default implementation of process_url/1 to:

1
2
3

defp process_url(url) do
  @endpoint <> url
end

and we’d have an invalid URL if we did.

The HTTPoison documentation is clear about this “overriding” feature:

HTTPoison.Base defines the following list of functions, all of which can be overridden (by redefining them)…
HTTPoison.Base docs

OK… but it’s not like this is OOP and looking into useing a module doesn’t explain anything about function overriding. So how does this work?

To better understand what’s going on, we need to dig a bit deeper. We know that HTTPoison.Base.__using__/1 macro is expanded at compile time when useing HTTPoison.Base, and if we look into it, we come across this line in the macro’s definition:

1	defoverridable Module.definitions_in(__MODULE__)

Module.definitions_in/1 is being passed the __MODULE__/0 pseudo variable, the evaluation of which seems to be listing all functions defined in HTTPoison.Base.

Pseudo variables return information about Elixir’s compilation environment and can only be read, never assigned to.
Pseudo variables

If we now take a look at the Kernel.defoverridable/1 macro, we’ll find out that this is what’s responsible for the overridability feature:

Makes the given functions in the current module overridable.
An overridable function is lazily defined, allowing a developer to override it.
Kernel.defoverridable/1 doc

It’s also interesting to note the example which comes with the doc for that macro:

defmodule DefaultMod do
  defmacro __using__(_opts) do
    quote do
      def test(x, y) do
        x + y
      end
      defoverridable [test: 2]
    end
  end
end

defmodule InheritMod do
  use DefaultMod
  def test(x, y) do
    x * y + super(x, y)
  end
end

Apart from noticing that defoverridable/1 takes a keyword list as an argument, implying that Module.definitions_in/1 returns a keyword list, we can see usage of super to call the default implementation of test in the example.

Lets see this super call in action by using it in our process_url/1:

defp process_url(url) do
  case url |> String.slice(0, 8) |> String.downcase do
    "http://" <> _ -> super url
    "https://" <> _ -> super url
    _ -> @endpoint <> url
  end
end

(Read up on the use of the pipe operator, |>, if you’re not familiar with it). Now, both of the following will give us a 200 status code:

iex(1)> Github.Gateway.get! "https://api.github.com/orgs/elixir-lang/repos"
# ...
iex(2)> Github.Gateway.get! "/orgs/elixir-lang/repos"
# ...

Of course, the same prefix check is happening in super in this case, but the point is it works as expected.

Poison

Moving on to JSON parsing with Poison. We’re already depending on it so if we iex -S mix, we can start exploring its basic usage (and how to get to specific bits of data):

iex(1)> %HTTPoison.Response{ body: body } = Github.Gateway.get! "/orgs/elixir-lang/repos"
# ...
iex(2)> i body
# ...
Data type
  BitString
Byte size
  58284
Description
  This is a string: a UTF-8 encoded binary. It's printed surrounded by
  "double quotes" because all UTF-8 encoded codepoints in it are printable.
iex(3)> parsed_body = Poison.Parser.parse! body
# ...
iex(4)> i parsed_body
# ...
Data type
  List
iex(5)> length parsed_body
12
iex(6)> parsed_body |> List.first |> Map.get("name")
"elixir"
iex(7)> :lists.nth(4, parsed_body)["name"]
"ex_doc"
iex(8)> elixir_repo = parsed_body |> Enum.at(0)
# ...
iex(9)> i elixir_repo
# ...
Data type
  Map
iex(10)> elixir_repo["full_name"]
"elixir-lang/elixir"
iex(11)> [_, _, _, %{ "full_name" => ex_doc_repo_full_name } | _] = parsed_body
# ...
iex(12)> ex_doc_repo_full_name
"elixir-lang/ex_doc"

I’m digressing a bit here to show off different ways in which you can manipulate the parsed data:

iex(1) is old news by now, we’ve already seen that we can do this to get the body of our HTTP request.
iex(2) is using the i iex helper function which shows us that body is a BitString.
iex(3) is parsing the body with Poison.Parser.parse!/2 and binding the result to parsed_body. Note: For more info on variable naming conventions and the significance of the “trailing bang” in Poison.Parser.parse!, you can read up on these in the online doc for naming conventions.
iex(5) shows us the length of parsed_body List via the built-in (i.e. imported by default in your modules and defined in the Kernel module) length function.
iex(6) is piping to List.first/1 to get the first element in parsed_body and then piping again to Map.get/3 to get the value for the "name" String key. Note that Map.get/3 has an optional 3rd argument (with a default value) and |> is supplying the first argument.
iex(7) is similarly getting the value for the "name" String key in the 4th element in parsed_body using Erlang’s lists:nth/2 function. Note that we cannot pipe parsed_body into this function as it takes the index for its first argument (as opposed to the List). Also note that the index for lists:nth/2 starts from 1 not 0.
iex(8) demonstrates another way of picking elements from a list using Enum.at/3.
iex(11) uses pattern matching to get to the value of the "full_name" key in the Map which is the 4th element in the parsed_body List.

So we’ve parsed some JSON into a list of maps and singled out bits of data from it. We’re now ready to write that data back out, or “encode” it back to JSON:

iex(13)> Poison.encode(elixir_repo)
{:ok,
"{\"teams_url\":\"https://api.github.com/repos/elixir-lang/elixir/teams\",
# ...
iex(14)> elem(Poison.encode(elixir_repo), 1)
"{\"teams_url\":\"https://api.github.com/repos/elixir-lang/elixir/teams\",
# ...
iex(15)> IO.puts elem(Poison.encode(elixir_repo), 1)
{"teams_url":"https://api.github.com/repos/elixir-lang/elixir/teams",
# ...
iex(16)> IO.puts elem(Poison.encode(elixir_repo, [pretty: 2]), 1)
{
  "teams_url": "https://api.github.com/repos/elixir-lang/elixir/teams",
  "branches_url": "https://api.github.com/repos/elixir-lang/elixir/branches{/branch}",
# ...
iex(17)> File.write("./elixir_repo.json", elem(Poison.encode(elixir_repo, [pretty: 2]), 1))
:ok
iex(18)> elixir_repo |> Poison.encode([pretty: 2]) |> elem(1) |> IO.puts
# ... same as iex(16)

iex(13) uses Poison.encode/2 without options (thus defaulting to an empty List) to encode the elixir_repo Map to JSON. This returns a tuple with :ok as the first element and the encoded JSON as the second.
iex(14) uses elem/2 to extract the JSON from the tuple.
iex(15) prints iex(14) to stdout.
iex(16) uses the [pretty: 2] option when encoding to print the JSON in a more readable fashion.
iex(17) does the same thing but writes to the "./elixir_repo.json" file instead of stdout.
iex(18) is just iex(16) using pipes.

While we’re at it, and since we have a large enough Map to demo it, consider the following:

iex(19)> Map.keys elixir_repo
["statuses_url", "git_refs_url", "issue_comment_url", "watchers", "mirror_url",
 "languages_url", "stargazers_count", "forks", "default_branch", "comments_url",
 "commits_url", "id", "clone_url", "homepage", "stargazers_url", "events_url",
 "blobs_url", "forks_count", "pushed_at", "git_url", "hooks_url", "owner",
 "trees_url", "git_commits_url", "collaborators_url", "watchers_count",
 "tags_url", "merges_url", "releases_url", "subscribers_url", "ssh_url",
 "created_at", "name", "has_issues", "private", "git_tags_url", "archive_url",
 "has_wiki", "open_issues_count", "milestones_url", "forks_url", "url",
 "downloads_url", "open_issues", "keys_url", "description", "contents_url",
 "language", "permissions", "contributors_url", ...]
iex(20)> elixir_repo |> Map.keys |> length
68
iex(21)> Inspect.Opts.__struct__
%Inspect.Opts{base: :decimal, binaries: :infer, char_lists: :infer, limit: 50,
 pretty: false, safe: true, structs: true, width: 80}
iex(22)> Inspect.Opts.__struct__.limit
50
iex(23)> elixir_repo |> Map.keys |> Enum.take(50)
["statuses_url", "git_refs_url", "issue_comment_url", "watchers", "mirror_url",
 "languages_url", "stargazers_count", "forks", "default_branch", "comments_url",
 "commits_url", "id", "clone_url", "homepage", "stargazers_url", "events_url",
 "blobs_url", "forks_count", "pushed_at", "git_url", "hooks_url", "owner",
 "trees_url", "git_commits_url", "collaborators_url", "watchers_count",
 "tags_url", "merges_url", "releases_url", "subscribers_url", "ssh_url",
 "created_at", "name", "has_issues", "private", "git_tags_url", "archive_url",
 "has_wiki", "open_issues_count", "milestones_url", "forks_url", "url",
 "downloads_url", "open_issues", "keys_url", "description", "contents_url",
 "language", "permissions", "contributors_url"]
iex(24)> IEx.configure(inspect: [limit: 70])
:ok
iex(25)> Map.keys elixir_repo
["statuses_url", "git_refs_url", "issue_comment_url", "watchers", "mirror_url",
 "languages_url", "stargazers_count", "forks", "default_branch", "comments_url",
 "commits_url", "id", "clone_url", "homepage", "stargazers_url", "events_url",
 "blobs_url", "forks_count", "pushed_at", "git_url", "hooks_url", "owner",
 "trees_url", "git_commits_url", "collaborators_url", "watchers_count",
 "tags_url", "merges_url", "releases_url", "subscribers_url", "ssh_url",
 "created_at", "name", "has_issues", "private", "git_tags_url", "archive_url",
 "has_wiki", "open_issues_count", "milestones_url", "forks_url", "url",
 "downloads_url", "open_issues", "keys_url", "description", "contents_url",
 "language", "permissions", "contributors_url", "pulls_url", "labels_url",
 "html_url", "svn_url", "issue_events_url", "notifications_url",
 "has_downloads", "compare_url", "full_name", "subscription_url",
 "assignees_url", "issues_url", "size", "has_pages", "fork", "updated_at",
 "branches_url", "teams_url"]

iex(19) prints out the keys in the elixir_repo Map - but note the trailing ellipsis after the "contributors_url" key, i.e. the shell doesn’t print all the keys since it is configured to print up to a limit of number of elements for certain data types, such as lists in this case.
iex(20) shows us how many keys the elixir_repo Map has (68).
iex(21) shows the current settings of Inspect.Opts which are used by the shell when inspecting values (and “inspecting values” is used when printing our expression results to the shell - and also, for e.g., when using the i helper).
iex(22) highlights the setting we’re currently interested in, the :limit field:

:limit - limits the number of items that are printed for tuples, bitstrings, and lists, does not apply to strings nor char lists, defaults to 50.
Inspect.Opts doc

iex(23) just shows that if we take 50 from the List of keys, we do indeed end up with all the keys printed in iex(19) (no ellipsis).
iex(24) is setting the inspection limit to 70 (by supplying a keyword list argument). More specifically, it is setting the IEx configuration for inspection via the :inspect keyword list element, whose value takes yet another keyword list made up of the Inspect.Opts fields we’ve just seen:

A keyword list containing inspect options used by the shell when printing results of expression evaluation. Default to pretty formatting with a limit of 50 entries.
See Inspect.Opts for the full list of options.
IEx.configure/1 :inspect option

iex(25) shows the result of this configuration. We are now able to print out all 68 keys.

You might want to read The .iex.exs file section in the IEx docs if you’re interested in setting IEx configuration on shell startup (or just setting up IEx with pre-bound variables etc…). Kudos to Gary Rennie for helping me find this when I was first looking for it.

Building an API with Streams

Back to our “intention” here. With all this background you should now be better equipped to digest most of the code in Drew’s original post. However, the Github.ResultStream module below is worth breaking down as it features a couple of things we haven’t yet discussed:

# lib/github_resultstream.ex
defmodule Github.ResultStream do
  alias Github.Gateway

  def new(url) do
    Stream.resource(
      fn -> fetch_page(url) end,
      &process_page/1,
      fn _ -> nil end
    )
  end

  defp fetch_page(url) do
    response = Gateway.get!(url)
    items = Poison.decode!(response.body)
    # TODO: fix access to "Link" header
    # Works "by accident" in v1.1 of Elixir
    # Fixed (i.e. doesn't work) in v1.2:
    links = parse_links(response.headers["Link"])

    {items, links["next"]}
  end

  def parse_links(nil) do
    %{}
  end

  def parse_links(links_string) do
    links = String.split(links_string, ", ")

    Enum.map(links, fn link ->
      [_,name] = Regex.run(~r{rel="([a-z]+)"}, link)
      [_,url] = Regex.run(~r{<([^>]+)>}, link)
      short_url = String.replace(url, Gateway.endpoint, "")

      {name, short_url}
    end) |> Enum.into(%{})
  end

  defp process_page({nil, nil}) do
    {:halt, nil}
  end

  defp process_page({nil, next_page_url}) do
    next_page_url
    |> fetch_page
    |> process_page
  end

  defp process_page({items, next_page_url}) do
    {items, {nil, next_page_url}}
  end
end

Stream.resource/3

To make sense of this, it’s probably best to just jump straight to the example given in the online docs:

# Stream.resource(start_fun, next_fun, after_fun)
# @spec resource((() -> acc), (acc -> {element, acc} | {:halt, acc}), (acc -> term)) :: Enumerable.t
Stream.resource(fn -> File.open!("sample") end,
                fn file ->
                  case IO.read(file, :line) do
                    data when is_binary(data) -> {[data], file}
                    _ -> {:halt, file}
                  end
                end,
                fn file -> File.close(file) end)

So Stream.resource/3 takes 3 functions, start_fun, next_fun, and after_fun.

The result of start_fun is fed as an argument to next_fun which is responsible for generating the stream’s values. In this case, that result is a process id (or PID):

1
2
3

# we're using the file we wrote in iex(17) above
iex(26)> File.open!("./elixir_repo.json")
#PID<0.64.0>

Every time a file is opened, Elixir spawns a new process.
File module doc

Note that the result of File.open!/2 is different from that of File.open/2 (also note that the second arg is optional in both cases):

1 2	iex(27)> File.open("./elixir_repo.json") {:ok, #PID<0.66.0>}

i.e. it’s wrapped in a tuple. You can read up on this in the “Trailing bang” section in the naming conventions doc. The File module‘s doc also mentions this in the “API” section.

This PID is called an io_device in the doc for the File module, and an io_device can be used as an argument to the IO module functions.

That is exactly what’s happening in next_fun above, which is passing file (bound to a PID), to IO.read/2 to read a line from the io_device:

iex(28)> file = v(26)
#PID<0.64.0>
iex(29)> IO.read(file, :line)
"{\n"
iex(30)> IO.read(file, :line)
"  \"teams_url\": \"https://api.github.com/repos/elixir-lang/elixir/teams\",\n"
iex(31)> IO.read(file, :line)
"  \"branches_url\": \"https://api.github.com/repos/elixir-lang/elixir/branches{/branch}\",\n"
iex(32)> is_binary IO.read(file, :line)
true

With the :line option, IO.read/2 will keep giving us binary data until we read the whole file, at which point it will return an :eof atom which will match our second clause in our next_fun‘s case expression. So from this we can see that next_fun either returns {[data], file} or {:halt, file} - i.e. either the next line in the file as a string wrapped in a list or :halt to mark the end of the stream, in both cases accompanied by file our accumulator (the thing by which we’re able to keep streaming values).

Successive values are generated by calling next_fun with the previous accumulator (the initial value being the result returned by start_fun) and it must return a tuple containing a list of items to be emitted and the next accumulator.

Why do we wrap the read line in a list?Stream.resource/3 doc

Finally, the after_fun is called with the accumulator (file) in order to give us an opportunity to clean up after ourselves, in this case, closing the file.

With that, we now know the control flow abstracted by Stream.resource/3. If we take another look at it’s usage in Github.ResultStream:

def new(url) do
  Stream.resource(
    fn -> fetch_page(url) end,
    &process_page/1,
    fn _ -> nil end
  )
end

the other things to note before diving into the meat of its implementation (process_page/1), are the use of Elixir partials to refer to process_page/1 to act as Stream.resource/3‘s next_fun, and the definition of a “do nothing” function for the after_fun (since we have nothing to clean up). Originally, the “do nothing” function was defined as: fn _ -> end in Drew’s post but the compiler gives a warning for this syntax as of Elixir 1.2, so we need to evaluate to nil (which is what used to happen anyway when no expression was given). Kudos to René Föhring for pointing this out.

[Kernel] Warn when right hand side of -> does not provide any expression
v1.2.0 release notes

The rest of Github.ResultStream

We’re good to move on… so let’s tackle the “TODO” comment in the snippet below:

defp fetch_page(url) do
  response = Gateway.get!(url)
  items = Poison.decode!(response.body)
  # TODO: fix access to "Link" header
  # Works "by accident" in v1.1 of Elixir
  # Fixed (i.e. doesn't work) in v1.2:
  links = parse_links(response.headers["Link"])

  {items, links["next"]}
end

Back when I was going through Drew’s post I was using Elixir v1.1.1 (there was no v1.2 yet). Lets try the following with v1.1.1:

iex(1)> headers = [{"Server", "GitHub.com"}]
[{"Server", "GitHub.com"}]
iex(2)> headers["Server"]
"GitHub.com"

When I started writing this post (a couple of days before the beginning of the year), v1.2 came out so I switched to the new hotness. Running the same thing in v1.2 gives:

iex(1)> headers = [{"Server", "GitHub.com"}]
[{"Server", "GitHub.com"}]
iex(2)> headers["Server"]
** (ArgumentError) the Access calls for keywords expect the key to be an atom, got: "Server"
    (elixir) lib/access.ex:64: Access.fetch/2
    (elixir) lib/access.ex:77: Access.get/3

At first, I thought something was broken in v1.2 but as José Valim points out, it looks like the behaviour in v1.1.1 was accidental! You can’t use that syntax on headers in this case because headers isn’t a keyword list or map. It’s a list of 2-element tuples but the first element in the tuple is not an atom (which would make it a keyword list) so this syntax won’t work. I thought I’d leave this bit in as I guess it’s “good to know”.

So we’re going to have to extract the “Link” header in some other way, which is just as well as it gives us the opportunity to look at a nice little trick I picked up from the blog post Writing assertive code with Elixir by José Valim:

defp fetch_page(url) do
  response = Gateway.get!(url)
  items = Poison.decode!(response.body)
  links_map = 
    response.headers
    |> Enum.find_value(fn({k, v}) -> k == "Link" && v end)
    |> parse_links
  {items, links_map["next"]}
end

First off, note the behaviour of the following:

iex(1)> true && "hello"
"hello"
iex(2)> "hello" && true
true
iex(3)> false && "hello"
false
iex(4)> "hello" && false
false
iex(5)> "hello" && "world"
"world"

Provides a short-circuit operator that evaluates and returns the second expression only if the first one evaluates to true (i.e., it is not nil nor false). Returns the first expression otherwise.
Kernel.&&/2 macro doc

This means that whenever k == "Link" evaluates to false, k == "Link" && v will always evaluate to false. Should it ever evaluate to true, then k == "Link" && v would evaluate to v

Enum.find_value/3 is similar to Enum.find/3, which I assume you’re already familiar with. The only difference is that instead of returning the element in the enumerable (list) for which the function you pass in evaluates to “not false or nil“, Enum.find_value/3 returns the function’s “truthy” value instead (where “truthy” is “not false or nil“).

(One gotcha that comes to mind would be if, for e.g., the list we’re querying has something like {“Link”, nil}. Since nil is also Enum.find_value/3‘s default return value if the function you pass in never evaluates to truthy, you wouldn’t be able to tell if the nil you get back from find_value/3 was a match or not).

This works well for our use case. Another approach suggested by Michał Muskała uses List.keyfind/4 and works just as well:

defp fetch_page(url) do
  response = Gateway.get!(url)
  items = Poison.decode!(response.body)
  links_map = 
    response.headers
    |> List.keyfind("Link", 0, {nil, nil})
    |> elem(1)
    |> parse_links
  {items, links_map["next"]}
end

I wanted to mention it anyway because it’s “good to know” the functions in the standard lib. For e.g. before finding out about List.keyfind/4, I would use Enum.find/3 but the former has a simpler API if you’re working with a list of tuples.

However, note that to get the same behaviour, you need to default to something like {nil, nil} in our case because if List.keyfind/4 doesn’t find the key and returns nil (by default), then elem/2 will throw an exception. We need to use elem/2 to get the value part of the “Link” header in this case as List.keyfind/4 gives us back the whole tuple.

So I’ll stick to Enum.find_value/3 for this case. You might want to read that Google Group thread for another interesting approach suggested by Ben Wilson which caters for multiple similar “keys” (i.e. first element of tuple) in the list. In our case, we know we’ll only have at most one “Link” header (possibly none), so it’s best not to use that approach here as the code would be longer (in both converting to a map and accessing the “Link” key which might not be there, and whose value would then be a list if it were there).

Ok ok… back on track:

#  defp fetch_page(url) do
#    response = Gateway.get!(url)
#    items = Poison.decode!(response.body)
#    links_map = 
#      response.headers
#      |> Enum.find_value(fn({k, v}) -> k == "Link" && v end)
#      |> parse_links
#    {items, links_map["next"]}
#  end
iex(1)> Github.ResultStream.parse_links("; rel=\"next\", ; rel=\"last\"")
%{"last" => "/organizations/9950313/repos?page=3",
  "next" => "/organizations/9950313/repos?page=2"}
iex(2)> v(1)["next"]
"/organizations/9950313/repos?page=2"
iex(3)> Github.ResultStream.parse_links(nil)
%{}
iex(4)> %{}["next"]
nil

With that, you should now know how our Stream.resource/3‘s start_fun works and what our first acc (for next_fun) will look like. Our first acc will be {items, nil | BitString} where items is the list of maps decoded by Poison (i.e. the repos data), and the second element in the tuple is either nil if there’s no more data to get through pagination, or the link to follow if there is.

Now, the only thing left is understanding our next_fun:

defp process_page({nil, nil}) do
  {:halt, nil}
end

defp process_page({nil, next_page_url}) do
  next_page_url
  |> fetch_page
  |> process_page
end

defp process_page({items, next_page_url}) do
  {items, {nil, next_page_url}}
end

First off, know that this is pattern matching at work again and that pattern matching happens top-to-bottom. So if we pass process_page/1 no data and no next link - {nil, nil} - the first clause will match and it’ll give us back {:halt, nil} (this will always happen at some point because this thing’s recursive and that’s the base case).

Let’s take the case of fetch_page("/orgs/elixir-lang/repos"), which as we saw before, doesn’t have enough data to need pagination - hopefully that will change soon ;)

In elixir-lang‘s case, it’s {items, nil} where items isn’t nil (thankfully!). This means the 3rd clause will match and our first evaluation of process_page/1 gives {items, {nil, nil}}.

Remember from our discussion of Stream.resource/3 above that the first element in the tuple returned by Stream.resource/3‘s next_fun, if not :halt, is a list of data, the data which our stream produces. We’re basically passing that data to Stream.resource/3 and using the second element in the tuple (the accumulator) to keep getting more data (until exhaustion), through next_fun - which is now invoked again with {nil, nil} and we already know what will happen next.

But what if we were querying something like “/orgs/nodejs/repos”? In that case, next_page_url would not be nil and our acc would be {nil, url}. The next time our stream is forced for more data, next_fun will be called with this acc which would match the 2nd clause. This will hit Github’s API for the next paginated data, and return {:halt, nil} if no data comes back, or {items, {nil, nil | next_page_url}} if there is data (note that it would be problematic if Github’s API would, for some weird reason (a.k.a bug), return no data but also include a “Link” header with a next URL to follow in its reply as we would basically end up in an infinite loop).

In the spirit of being explicit about anything that might be ambiguous (a.k.a might as well go all the way), if Github comes back with data and a “Link” header with a next URL to follow, then when our stream is forced for more data, process_page/1 will be called with {nil, next_page_url}

If Github returns no “Link” header with a next URL to follow, then when our stream is next forced for more data, process_page/1 will be called with {nil, nil}, but in both cases the data Github returned is fed back to Stream.resource/3 via items.

Test drive

The Github module below exposes a clean API to our users:

# lib/github.ex
defmodule Github do
  alias Github.ResultStream

  def repos(organization) do
    ResultStream.new("/orgs/#{organization}/repos")
  end
end

Below taking this thing for a spin, go ahead and add IO.puts("Url: #{url}") to Github.Gateway.process_url/1 so we can confirm how many requests are being made (as we’ve seen before, HTTPoison uses this function internally):

iex(1)> repo_stream = "nodejs" |> Github.repos |>
...(1)>   Stream.map(fn repo_map -> {repo_map["full_name"], repo_map["forks"]} end)
#Stream<[enum: #Function<46.94256892/2 in Stream.resource/3>,
 funs: [#Function<29.94256892/1 in Stream.map/2>]]>
iex(2)> repo_stream |> Enum.take(4)
Url: /orgs/nodejs/repos
[{"nodejs/http-parser", 641}, {"nodejs/node-v0.x-archive", 8508},
 {"nodejs/node-gyp", 354}, {"nodejs/readable-stream", 67}]
iex(3)> repo_stream |> Enum.take(40)
Url: /orgs/nodejs/repos
Url: /organizations/9950313/repos?page=2
[{"nodejs/http-parser", 641}, {"nodejs/node-v0.x-archive", 8508},
 {"nodejs/node-gyp", 354}, {"nodejs/readable-stream", 67},
 {"nodejs/node-addon-examples", 140}, {"nodejs/nan", 180},
 {"nodejs/nodejs.org-archive", 83}, {"nodejs/build", 26},
 {"nodejs/roadmap", 34}, {"nodejs/build-containers", 5}, {"nodejs/node", 2382},
 {"nodejs/iojs.org", 138}, {"nodejs/logos", 40}, {"nodejs/docker-node", 72},
 {"nodejs/build-container-sync", 2}, {"nodejs/doc-tool", 7},
 {"nodejs/docker-iojs", 28}, {"nodejs/tracing-wg", 8}, {"nodejs/nodejs-es", 3},
 {"nodejs/nodejs-ko", 31}, {"nodejs/nodejs-nl", 2}, {"nodejs/nodejs-de", 4},
 {"nodejs/nodejs-pt", 6}, {"nodejs/nodejs-ru", 14}, {"nodejs/nodejs-fr", 6},
 {"nodejs/nodejs-da", 1}, {"nodejs/nodejs-hi", 2}, {"nodejs/nodejs-ja", 9},
 {"nodejs/nodejs-no", 5}, {"nodejs/nodejs-hu", 4}, {"nodejs/nodejs-he", 1},
 {"nodejs/nodejs-it", 1}, {"nodejs/nodejs-sv", 0}, {"nodejs/nodejs-tr", 4},
 {"nodejs/nodejs-ka", 0}, {"nodejs/nodejs-mk", 0}, {"nodejs/nodejs-pl", 0},
 {"nodejs/nodejs-el", 1}, {"nodejs/nodejs-id", 6}, {"nodejs/nodejs-cs", 0}]

As we saw with curl + jq in the How to fetch Github organization repos and their pagination, Github will only ever give us a max of 30 repos per request, so when we Enum.take(40) in iex(2) above, we get the second request’s URL printed out as our stream needs to paginate. In comparison, only one request ever happens in iex(1) which only ever takes 4 repos from the stream.

How to install Elixir with asdf

asfd is an extendable version manager for a couple of platforms (currently, Ruby, Node.js, Elixir, Erlang, and Lua) i.e. you can use it to install, uninstall, and change between different versions of the supported platforms.

To install it on your system, simply git clone https://github.com/HashNuke/asdf.git ~/.asdf and source it from a startup script like .bashrc or .bash_profile or similar, by running:

echo '. $HOME/.asdf/asdf.sh' >> ~/.bash_profile

i.e. append . $HOME/.asdf/asdf.sh to ~/.bash_profile.

To get started using Elixir, you then need to install both Erlang and Elixir on your system using the respective asdf plugin for each:

$ asdf plugin-add erlang https://github.com/HashNuke/asdf-erlang.git
# ...
$ asdf list-all erlang
18.1
# ...
$ asdf install erlang 18.1
# ...
$ asdf plugin-add elixir https://github.com/HashNuke/asdf-elixir.git
# ...
$ asdf list-all elixir
# 1.1.1
# .... Note: don't worry if 1.2.0 is not listed, install it anyway (the list is hard-coded)
$ asdf install elixir 1.2.0
# ...

One last thing I’d like to point out about asdf is its .tool-versions feature which you can use to set default versions of the tools you use, both globally (by using it in your home directory), as well as on a per-project basis.

Creating a Ruby on Rails Vagrant box

2015-12-26T00:00:00.000Z

Intro

In this post we’ll see how to create a virtual machine running Ubuntu which has everything installed to start developing Rails applications.

Prerequisites

You’ll need to have VirtualBox and Vagrant installed before starting (if you intend to follow along). I’m using VirtualBox version 5.0.10 r104061 and Vagrant version 1.7.4.

Starting up a VM with Vagrant

Start off by creating an empty directory, e.g. rails-box, and cd into it. This is where we’ll be building our template box containing everything we need to start writing a new Rails application. After building it, we’ll be able to install the template on our system and use it whenever we want an Ubuntu VM pre-installed with the software we need to start writing Rails apps.

In order to kick off the process, go ahead and run vagrant init ubuntu/trusty64 which will give us a Vagrantfile specifying which box we want to use (in our case, it’s ubuntu/trusty64). A box whose name is in the format vendor/name means Vagrant will look for this box on Hashicorp‘s servers if it fails to find it on your host machine.

Note, if you don’t fancy all the comments in the generated Vagrantfile, feel free to delete the file and generate it again using the -m flag: vagrant init -m ubuntu/trusty64.

With our Vagrantfile in place, go ahead and start the VM with vagrant up. This can take quite some time if you don’t already have a template for ubuntu/trusty64 on your machine. In this case, Vagrant will need to download the box from Hashicorp’s servers and store it in your $VAGRANT_HOME which by default is ~/.vagrant.d.

Downloading the box will be the longest part of vagrant up. After which, it also needs to extract and install the template box in VirtualBox. Apart from the stdout log output, you’ll be able to see when this is done in VirtualBox’s GUI Manager as another entry for the new box will be listed (with status “Running”).

Installing software

Now that Ubuntu’s running, go ahead and log into the box via ssh with: vagrant ssh (run this in the same directory containing your Vagrantfile). Once logged in, if you run ruby -v you’ll see that it’s a bit old (for me it’s ruby 1.9.3p484).

So the first thing we’ll do is upgrade our Ruby. There’s a couple of ways of doing this. In the past, I remember (successfully) using RVM for this (and more). As I’m just returning to Ruby land after 2 years of meddling in other things, I’m following along Rails 4 in Action, so we’ll be using the approach suggested there.

To get started, we need a copy of ruby-install, which you can get by running the following in your ssh session at the command prompt:

1	~$ wget -O ruby-install-0.5.0.tar.gz https://github.com/postmodern/ruby-install/archive/v0.5.0.tar.gz

I’m using the latest at the time of writing but feel free to get any updated version. You can then simply extract and install:

~$ tar -xzvf ruby-install-0.5.0.tar.gz
...
~$ cd ruby-install-0.5.0/
~/ruby-install-0.5.0$ sudo make install

You should now have ruby-install on your PATH and be able to execute it at the command prompt. Go ahead and install the version of Ruby you want with: ruby-install ruby 2.2.1 (and yes, this does take a while). When it’s done, you’ll get something like:

>>> Successfully installed ruby 2.2.1 into /home/vagrant/.rubies/ruby-2.2.1

Cool, but if you ruby -v, you’ll still see the old version. First off, since this is going to be a template box, it’s probably best to clean up after ourselves now that ruby-install is, well, installed:

1 2	~/ruby-install-0.5.0$ cd ~ ~$ rm -rf ruby-install-0.5.0 src ruby-install-0.5.0.tar.gz

Now, moving on to setting which version of Ruby we want to use. For this we’ll use chruby, and the process is quite similar:

~$ wget -O chruby-0.3.9.tar.gz https://github.com/postmodern/chruby/archive/v0.3.9.tar.gz
~$ tar -xzvf chruby-0.3.9.tar.gz
~$ cd chruby-0.3.9/
~/chruby-0.3.9$ sudo make install
~/chruby-0.3.9$ cd ..
~$ rm -rf ./chruby-0.3.9 chruby-0.3.9.tar.gz

After installing chruby in your system, you’ll need to source it from one of your startup scripts:

1	~$ echo -e "\nsource /usr/local/share/chruby/chruby.sh" >> ~/.bashrc

You can also source the auto.sh script if you want to enable the automatic switching of Ruby versions based off of the presence/contents of a .ruby-version file:

1	~$ echo "source /usr/local/share/chruby/auto.sh" >> ~/.bashrc

This isn’t strictly necessary, but why not. It comes in handy if you’re working on a project which requires the use of a particular version of Ruby. You could save a .ruby-version file with that particular version as content (in your project’s root). Now, every time you cd into that project, you don’t have to remember to switch to that particular version of Ruby as it’s done automatically for you.

Moving on, you can exit your ssh session and log in again with vagrant ssh to actually source these files. You now have chruby on your PATH and running it will list out the Ruby versions you have installed. To finally actually use the version we previously installed, run: chruby ruby-2.2.1.

Now you can verify we’re using that version of Ruby with ruby -v. Since you don’t want to manually have to stay doing this every time you log in, add this to your .bashrc file or make use of the auto.sh feature we enabled:

1
2
3

~$ echo "chruby ruby-2.2.1" >> ~/.bashrc
~$ # and/or
~$ sudo sh -c 'echo "ruby-2.2.1" >> /.ruby-version'

It’s easiest for us to stick with the .bashrc approach but if you opt to use .ruby-version, put it in the root directory as shown above. We’ll later be using Rails in /vagrant which is outside /home/vagrant so a .ruby-version there wouldn’t be picked up by chruby.

Now it’s Rails’s turn:

1	~$ gem install rails -v 4.2.1

Confirm everything’s in order with rails -v.

Again, following along the Rails 4 in Action book, install the following header files via apt-get:

1	~$ sudo apt-get update -y && sudo apt-get install -y libsqlite3-dev libpq-dev libmysqlclient-dev

Installing a Javascript runtime

The last piece of software we’ll need to install is a Javascript runtime. I’ll be going with Node.js as that’s what I’m more familiar with. However, if I apt-cache policy nodejs to see which version of Node I’d get if I were to apt-get install it, I get back 0.10.25… not exactly the latest and greatest.

As we’ve done with Ruby, we’ll use tools which will take care of installing and managing our versions of Node. I’m personally somewhat familiar with 2 of these: nvm and asdf, and I’d like to quickly cover basic usage of both here. However, nvm is the more mature of the two from its Github stats (and through personal usage). It is also more specialized in that it’s only concerned with catering for Node.js.

asdf’s appeal is in its pluggability, as playfully put in the ballad-of-asdf.md.

nvm

To install nvm run the following (you will find the URL to the latest version of nvm in its Github README.md):

1	~$ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.29.0/install.sh \| bash

Log out and back into a new ssh session and you should be good with nvm on your PATH. To install Node via nvm, you’d run something like:

1	~$ nvm install 5.3.0

You can find the latest version number for Node from here. You should be able to node -v and get the version now, but if at this point you were to log out and back in to the machine, your session wouldn’t have Node on the PATH. In order to fix this, create a .nvmrc file like so:

1	~$ echo "5.3.0" >> ~/.nvmrc

asdf

If you’ve followed the nvm instructions, you’re good to go. You could install asdf anyway, and even install versions of Node with it - i.e. you can have both nvm and asdf on your system and then choose which one you want to use by sourcing the tool of choice from your .bashrc. Uninstalling either of them is simply a matter of not sourcing in .bashrc and deleting their home directory (~/.nvm or ~/.asdf).

To get started, install git, clone asdf, and source its script file from your init script:

1
2
3

~$ sudo apt-get install git -y
~$ git clone https://github.com/HashNuke/asdf.git ~/.asdf
~$ echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bashrc

Log out and back in. You’ll then need to install the Node.js plugin with:

1	~$ asdf plugin-add nodejs https://github.com/HashNuke/asdf-nodejs.git

and then Node itself with:

1	~$ asdf install nodejs 5.3.0

The same file based setting of version number can be achieved with:

1	~$ echo "nodejs 5.3.0" >> ~/.tool-versions

Packaging up our box

At this point, our box seems to be well equipped for Rails development so we’ll create a template out of it.

In your host OS (in Vagrantfile dir), stop the running VM and package it up with:

1 2	$ vagrant halt $ vagrant package --output rails.box

When it’s done, you’ll get rails.box, a template box from which you can later re-create the exact same VM with all the software we’ve installed in it.

This box isn’t yet installed in Vagrant - something you can verify with vagrant box list. You can install it with:

1	$ vagrant box add rails ./rails.box

which will add the generated box under the name rails.

To use the installed box, go to an empty directory (where you intend to start developing your Rails app), and initialize + start it with:

1 2	~/rails-projects$ vagrant init -m rails ~/rails-projects$ vagrant up

If you now ssh into the new box with vagrant ssh you can verify that you have all the software we’ve previously installed.

Accessing Rails from the host OS

So far so good. We can now go on any machine, install Virtual Box, Vagrant, and the box we just created, and get a VM with all the software we need to start a Rails project.

That said, lets kick off a vanilla Rails app to demo the first issue you’ll have with this setup, namely, accessing your Rails app from a browser running on your host OS.

In your guest OS, go to the default shared directory on any stock Vagrant box, /vagrant. In it you will find the Vagrantfile we generated with vagrant init -m rails above:

~$ cd /vagrant/
/vagrant$ cat Vagrantfile
Vagrant.configure(2) do |config|
  config.vm.box = "rails"
end

Working here allows us to use our guest OS to execute any Rails commands, and our host OS to author the app with the software we already have installed.

Go ahead and generate a new Rails app with:

1	/vagrant$ rails new tmpapp

and then start it with:

1 2	/vagrant$ cd tmpapp /vagrant/tmpapp$ rails server

You will notice that Rails 4.2.1 starts up on http://localhost:3000. localhost is the guest OS’s loopback address, so there’s no way you’re going to access that endpoint from a browser on your host OS. Even if you were to start Rails listening on 0.0.0.0 for “listen on all IP addresses on this machine”, you still wouldn’t be able to access Rails from a browser on your host OS because the two aren’t on the same network.

To get around this, you could stay port forwarding the ports you’re interested in (say, 8080 on the host maps to 3000 on the guest), but that gets annoying really quickly if you later need to open other ports. I’ll opt for putting the two on the same private network.

The configuration required to tell Vagrant to set up this private network for us is actually commented out when you generate a Vagrantfile without the -m option (go ahead and vagrant init in an empty directory to generate it again). The line you want uncommented is:

1
2
3

# Create a private network, which allows host-only access to the machine
# using a specific IP.
config.vm.network "private_network", ip: "192.168.33.10"

You could go for a public_network instead, in which case your VM would be accessible from other devices on the network (not just your host), but I’m not sure whether you can set a static IP for the machine with this option. As you can see below, no IP address is given for this option (this means you’d have to stay checking which IP address is assigned to the VM every time you boot it):

# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network "public_network"

After adding the private network, our Vagrantfile looks like this:

Vagrant.configure(2) do |config|
  config.vm.box = "rails"
  config.vm.network "private_network", ip: "192.168.33.10"
end

To put this change into effect, you’ll need to restart the VM with vagrant reload (run it in the same directory containing your Vagrantfile in your host OS).

After logging in again with vagrant ssh, running ifconfig should show us the new IP address. In fact, you should be able to successfully ping 192.168.33.10 from the host OS.

If you now cd into the Rails app, you can start Rails listening on that IP address (or on 0.0.0.0) with: rails s -b 192.168.33.10. After doing this, you should be able to access Rails from your host OS by browsing to http://192.168.33.10:3000/.

One last thing. If you read the Rails log output after hitting Rails from your host OS’s browser, you’ll find:

Cannot render console from 192.168.33.1! Allowed networks: 127.0.0.1, ::1, 127.0.0.0/127.255.255.255

Our guest’s IP address is 192.168.33.10, and our host’s is (automatically set to) 192.168.33.1 i.e. the host’s IP address is the same but with 1 for the final octet of the address. If you ifconfig | grep -C 2 .33.1 on the host OS (assuming you’re on a Unixy host), you’ll verify this (it’s probably ipconfig on Windows). Or you could just ping 192.168.33.1 from the guest.

In any case, Rails seems to be complaining that we’re accessing it from the host (192.168.33.1), but it renders the console just fine in our browser so that message is just noise to us.

To get rid of it, you can follow the advise in this SO question, i.e. add this to config/environments/development.rb:

1	config.web_console.whitelisted_ips = '192.168.33.1'

Unfortunately, this is something you’ll need to remember to do everytime you start a new Rails project with this setup.

Vanilla CSS grid

2015-11-08T00:00:00.000Z

Intro

Pulling in a CSS framework such as Bootstrap, or even just it’s grid system, can be overkill when all you want is a simple page with a couple of columns in it.

In this post we’ll be using plain CSS to get some responsive columns on our page, effectively writing a minimal grid system.

Setting up the markup

To follow along, you can start with a basic page as shown below:


<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width">
    <title>title>
    <link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/meyer-reset/2.0/reset.css">
    <link rel="stylesheet" href="css/styles.css">
  head>
  <body>
    
  body>
html>

After pulling in reset.css (to remove inconsistencies in default styles across different browsers), we’re simply linking to our custom styles in styles.css.

Next, you need to decide on a class name to give to your containing element. Frameworks like Bootstrap use the class name “container” so it’s best to choose another name just in case you later decide to use such a framework. I’ll be using “grid-container”:

1 2	<div class="grid-container"> div>

Inside the grid container, we’ll be adding rows as shown in the example below:

<div class="row">
    <div class="col-xsm"><p>col-xsm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
    <div class="col-xsm"><p>col-xsm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
    <div class="col-xsm"><p>col-xsm 3. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
    <div class="col-xsm"><p>col-xsm 4. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
div>
<div class="row">
    <div class="col-sm"><p>col-sm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
    <div class="col-sm"><p>col-sm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
div>

Here we have 2 rows, the first made up of 4 col-xsm columns, while the second, of 2 col-sm columns.

Styles

The first thing we’ll do in our styles.css is target the grid container, as well as anything within it, and set the box-sizing property to border-box:

.grid-container, .grid-container * {
  -webkit-box-sizing: border-box;
  -moz-box-sizing: border-box;
  box-sizing: border-box;
}

This will include the padding and border in each element’s total width and height (so later, when we’ll be sepecifying widths in percentages, padding and boarder will be included).

Targetting the grid container again, we can center it by setting it’s left and right margins to auto. I’m also setting it’s max width to 750px here as I’ll be using these styles on this page and that’s the max width of my post content. Feel free to increase this for your purposes. Setting the width to 100% means the grid will take up all available space until the viewport reaches a width of 750px, at which point the grid’s width will stop expanding:

.grid-container {
  width: 100%;
  margin: 0 auto; 
  max-width: 750px;      
}

We’ll now target all our columns using an attribute selector on classes which begin with “col-“. As we’re going to start styling with the smallest viewport in mind, all our columns will be 100% wide (except for col-xsm which will be 50%). Floating our columns to the left will allow us to lay them out as columns on the page, and adding some padding is in order after resetting our styles:

[class^='col-'] {
    float: left; 
    width: 100%;
    padding: 10px;
}
.col-xsm { width: 50%; }

Before our first preview, lets go ahead and add an outline around all our elements to get a better idea of the spacing between them:

1
2
3

.outline, .outline *{
  outline: 1px solid #d6d6d6; 
}

At this point, given the HTML below:

<div class="grid-container outline">
  <div class="row-noclearfix">
      <div class="col-xsm"><p>col-xsm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Sit facilis nulla accusamus veniam at, corrupti cupiditate! Eveniet aliquam aspernatur ipsum vitae molestias similique eum perferendis sunt sit minus. Rem cupiditate optio rem omnis.p>div> 
      <div class="col-xsm"><p>col-xsm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
  div>
  <div class="row-noclearfix">
      <div class="col-xsm"><p>col-xsm 3. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
      <div class="col-xsm"><p>col-xsm 4. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
  div>
  <div class="row-noclearfix">
      <div class="col-sm"><p>col-sm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
      <div class="col-sm"><p>col-sm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
  div>
div>

we get something like this:

col-xsm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Sit facilis nulla accusamus veniam at, corrupti cupiditate! Eveniet aliquam aspernatur ipsum vitae molestias similique eum perferendis sunt sit minus. Rem cupiditate optio rem omnis.

col-xsm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-xsm 3. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-xsm 4. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-sm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-sm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

As you can see, the second row starting with col-xsm 3 is not in place. The reason this is happening is becuase we have not cleared our floats. So in order to fix this, we can add the following clearfix hack:

.row:before, 
.row:after {
    content:" ";
    display: table;
}
.row:after {
    clear:both;
}

After changing the row classes back to “row” in the example above, we get the desired effect:

col-xsm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-xsm 3. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-xsm 4. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-sm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-sm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

Adding media queries

So far we’ve been styling with mobile in mind, which is why we’ve specified all our columns to be 100% wide except for our extra small ones (col-xsm) which are 50% wide. It’s now time to change that, but before doing so, we need to figure out at which breakpoints we want our column widths to change. I’ll be using min-widths of 480px and 768px:

@media screen and (min-width: 480px) {
  .grid-container.r {
    background: tomato;
  }
}

@media screen and (min-width: 768px) {
  .grid-container.r {
    background: tan;
  }
}

This means something like: “when the browser width is greater than or equal to 480px, apply the styles in the first media query; then when it gets greater or equal to 768px, override/add with the styles in the second media query”.

You will notice I’m applying the styles only to elements which have both the class of grid-container as well as the class r on them. I’m doing this in order to leave previous examples on this page uneffected, and I’ll be doing the same when making our columns responsive as we go forward. Obviously, you should remove the .r part in your selectors and avoid littering your HTML with these extra classes on your own pages.

One other thing to note as we go forward. When resizing Chrome with the developer tools open, you will notice “width x height” dimensions given in the top right of the browser. Make sure you’re at 100% zoom of the page if you want the width shown in Chrome to match up to the breakpoints used in your media queries.

Ok, so lets quickly check that our media queries are working:

1
2
3

<div class="grid-container r">
  <p>Sit autem iure eum voluptates possimus. Voluptas illo quod dolorem?p>
div>

Sit autem iure eum voluptates possimus. Voluptas illo quod dolorem?

At a viewport of over 768px wide, you should see the lorem ipsum above with a tan background. At over 480px but less than 768px it should be tomato, and at under 480px neither will apply leaving us back with our mobile styles.

With that in place, we can go ahead and fill in our desired column widths:

@media screen and (min-width : 480px) {
  .grid-container.r {
    background: tomato;
  }
  .col-xsm.r  { width: 33.33%;  }
  .col-sm.r   { width: 50%;     }
  .col-md.r   { width: 83.33%;  }
  .col-lg.r   { width: 100%;    }
  .col-xlg.r  { width: 100%;    }

}

@media screen and (min-width: 768px) {
  .grid-container.r {
    background: tan;
  }
  .col-xsm.r  { width: 25%;  }
  .col-sm.r   { width: 33.33%;     }
  .col-md.r   { width: 50%;  }
  .col-lg.r   { width: 83.33%;    }
  .col-xlg.r  { width: 100%;    }
}

We’re setting col-xsm to take up 1/3 of my max-width of 750px (my blog post width) when the viewport’s width is between 480px and 767px inclusive. It’s going to change to be 25% that width when the viewport is 768px wide or wider, and it’s going to go back to 50% when the viewport is 479px wide or less.

Try changing the browser’s width to see the result in the following:

<div class="grid-container r">
  <div class="row">
      <div class="col-xsm r"><p>col-xsm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Sit facilis nulla accusamus veniam at, corrupti cupiditate! Eveniet aliquam aspernatur ipsum vitae molestias similique eum perferendis sunt sit minus. Rem cupiditate optio rem omnis.p>div> 
      <div class="col-xsm r"><p>col-xsm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
  div>
  <div class="row">
      <div class="col-sm r"><p>col-sm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Consectetur alias quos minima quae at! Consequuntur repellendus laboriosam laborum fugiat similique pariatur. Quidem repellat necessitatibus sed saepe voluptatum obcaecati modi maxime cum vero autem. Odit molestiae labore laudantium odit!p>div> 
      <div class="col-sm r"><p>col-sm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?p>div> 
  div>
  <div class="row">
      <div class="col-md r"><p>col-md 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Consectetur alias quos minima quae at! Consequuntur repellendus laboriosam laborum fugiat similique pariatur. Quidem repellat necessitatibus sed saepe voluptatum obcaecati modi maxime cum vero autem. Odit molestiae labore laudantium odit!p>div> 
  div>
  <div class="row">
      <div class="col-lg r"><p>col-lg 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Consectetur alias quos minima quae at! Consequuntur repellendus laboriosam laborum fugiat similique pariatur. Quidem repellat necessitatibus sed saepe voluptatum obcaecati modi maxime cum vero autem. Odit molestiae labore laudantium odit!p>div> 
  div>
  <div class="row">
      <div class="col-xlg r"><p>col-xlg 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Consectetur alias quos minima quae at! Consequuntur repellendus laboriosam laborum fugiat similique pariatur. Quidem repellat necessitatibus sed saepe voluptatum obcaecati modi maxime cum vero autem. Odit molestiae labore laudantium odit!p>div> 
  div>
div>

col-xsm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-sm 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Consectetur alias quos minima quae at! Consequuntur repellendus laboriosam laborum fugiat similique pariatur. Quidem repellat necessitatibus sed saepe voluptatum obcaecati modi maxime cum vero autem. Odit molestiae labore laudantium odit!

col-sm 2. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi?

col-md 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Consectetur alias quos minima quae at! Consequuntur repellendus laboriosam laborum fugiat similique pariatur. Quidem repellat necessitatibus sed saepe voluptatum obcaecati modi maxime cum vero autem. Odit molestiae labore laudantium odit!

col-lg 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Consectetur alias quos minima quae at! Consequuntur repellendus laboriosam laborum fugiat similique pariatur. Quidem repellat necessitatibus sed saepe voluptatum obcaecati modi maxime cum vero autem. Odit molestiae labore laudantium odit!

col-xlg 1. Adipisicing delectus nisi sequi placeat amet! Quasi ullam nulla iure saepe velit eos doloremque facere modi? Consectetur nulla possimus sequi? Consectetur alias quos minima quae at! Consequuntur repellendus laboriosam laborum fugiat similique pariatur. Quidem repellat necessitatibus sed saepe voluptatum obcaecati modi maxime cum vero autem. Odit molestiae labore laudantium odit!

War in jar with Maven

2015-04-28T00:00:00.000Z

Intro

The following is one way you could go about packaging a war file in a jar file with Maven. You might want to do this if you’re loading a war file from the classpath programmatically.

For instance, assuming you had myjar.jar on the classpath and myjar.jar contained mywar.war, then you could refer to this war like so: classpath://mywar.war. Simply adding the directory containing your war file on the classpath (as you would to pick up *.class files) would not be enough to get a similar result.

By example

To get something to work with, we can make use of the maven-archetype-webapp archetype:

$ mvn archetype:create  \
  -DgroupId=com.tmp \
  -DartifactId=warinjar \
  -DarchetypeArtifactId=maven-archetype-webapp
$ cd warinjar

Open up pom.xml in the generated project and change the packaging to jar:

1	<packaging>jarpackaging>

The final ingredient is to get the maven-war-plugin in on the action:

...
<build>
    <finalName>warinjarfinalName>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.pluginsgroupId>
            <artifactId>maven-war-pluginartifactId>
            <version>2.6version>
            <configuration>
                <outputDirectory>${project.build.directory}/classesoutputDirectory>
            configuration>
            <executions>
                <execution>
                    <id>build-war-in-classesid>
                    <phase>prepare-packagephase>
                    <goals>
                        <goal>wargoal>
                    goals>
                execution>
            executions>
        plugin>
    plugins>
build>
...

By setting the output directory to ${project.build.directory}/classes, this plugin will generate warinjar.war in our target/classes directory, the contents of which are picked up when packaging our project (since we set the packaging to jar above).

Running mvn clean package will clean out previous build (by deleting the target directory). It will also execute maven-war-plugin‘s war goal since we’re executing it in the prepare-package lifecycle phase (which happens right before the package phase in Maven’s default lifecycle).

This gives us target/classes/warinjar.war which then gets picked up when producing target/warinjar.jar (have a look at the jar’s contents).

Of course, you could add another execution to also generate a war outside the jar:

...
<execution>
    <id>build-warid>
    <phase>packagephase>
    <configuration>
        <outputDirectory>${project.build.directory}outputDirectory>
        <warName>waroutjarwarName>
    configuration>
    <goals>
        <goal>wargoal>
    goals>
execution>
...

Note that we need to overwrite the plugin’s configuration for this execution. Have a look at the documentation page for default values.

Basically, we’re overwriting the default outputDirectory in our plugin’s configuration, and so, the execution with id build-war-in-classes will output ${project.build.finalName}.war (i.e. warinjar.war) to ${project.build.directory}/classes.

Had we not overwritten the configuration in the plugin’s execution with id build-war, we would have basically overwritten target/classes/warinjar.war with itself.

The final pom file looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0modelVersion>
    <groupId>com.tmpgroupId>
    <artifactId>warinjarartifactId>
    <packaging>jarpackaging>
    <version>1.0-SNAPSHOTversion>
    <build>
        <finalName>warinjarfinalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.pluginsgroupId>
                <artifactId>maven-war-pluginartifactId>
                <version>2.6version>
                <configuration>
                    <outputDirectory>${project.build.directory}/classesoutputDirectory>
                configuration>
                <executions>
                    <execution>
                        <id>build-war-in-classesid>
                        <phase>prepare-packagephase>
                        <goals>
                            <goal>wargoal>
                        goals>
                    execution>
                    <execution>
                        <id>build-warid>
                        <phase>packagephase>
                        <configuration>
                            <outputDirectory>${project.build.directory}outputDirectory>
                            <warName>waroutjarwarName>
                        configuration>
                        <goals>
                            <goal>wargoal>
                        goals>
                    execution>
                executions>
            plugin>
        plugins>
    build>
project>

Sequential control flow with Lo-Dash or Underscore.js

2015-03-08T00:00:00.000Z

Intro

This post is based on Simple async with only underscore by Adrian Rossouw. Yes, I did read the following disclaimer regarding this approach:

I don’t recommend this approach much anymore, because I have come to the conclusion that the async.js module is by far the best tool for the job
Simple async with only underscore

It’s hard to disagree with that. Async.js is specifically designed to cater for asynchronous control flow. However, I think there is value in using the approach suggested by Adrian when you’re already depending on Lo-Dash/Underscore.js and do not want to bring in a separate library to handle a one-off control flow need.

In this post I’ll be explaining the suggested solution in more detail. But before doing that, I will give some background so you might want to skip over to The approach if you’re already familiar with the need for asynchronous control flow in Node.js / event loop environments.

The need

Sync vs Async in a single threaded environment

So what’s this all about? Lets set up an example. In an empty directory do the following:

npm i inquirer
vim index.js

var inquirer = require("inquirer");

var questions = [
  {
    type: "input",
    name: "username",
    message: "Enter username"
  },
  {
    type: "password",
    name: "password",
    message: "Enter password"
  },
];

inquirer.prompt(questions, function( answers ) {
  console.log("answers were:", answers);
});

console.log("I want this to be done after prompt");

Nevermind the duplicate “Enter username” prompt - notice that the final console.log statement executed before inquirer.prompt was done.

The code above assumes that expressing the two function invocations in sequence implies sequential execution. In other (multi-threaded) environments, where threads block on I/O, this is a reasonable assumption to make. In these environments, context switching allows one thread to block on I/O, switch over to another thread to keep processing, and switch back when data to process becomes available from the I/O source in question.

In a single threaded environment, however, you generally don’t want to block. For example, imagine uploading a couple of files from a web application. You wouldn’t want to “stop the world” while this data is still in transit. Instead, you would want to keep processing - for e.g. the user’s input when they decide to stop the upload.

So we can agree that blocking is not what we generally want to do in a single threaded environment. In the user prompt example above, you do not expect inquirer.prompt to “stop the world” while it fetches data from the user via process.stdin. The code inquirer.prompt uses to get data from the user does so in an asynchronous manner. It could do the same thing in a synchronous manner instead, but that is not good manners in a single threaded environment. If inquirer.prompt fetched its data synchronously and returned it, then the code above would work as expected but inquirer is not that selfish. It will wait on the stack for data to come in, allowing the processor to go do something useful in the meantime.

For instance, if we replaced user prompting with a function which reads from disk in a synchronous manner then we get something like this:

var fs = require("fs");
var path = require("path");

var data = fs.readFileSync(path.resolve(__dirname, __filename), "utf-8");

console.log("I want this to be done after readFileSync\n", data);

To summarize, the key points are:

In a single threaded environment asynchronous I/O becomes essential if you want concurrency.
If you have statements which execute asynchronously, the order in which you write your statements does not imply the order in which these statements are executed.

OK, so we know asynchronous I/O is important here, but the order of execution is something we need control over as sometimes we don’t really care, but sometimes there’s no way around it. This means that we need to be able to express sequential execution when it matters - i.e. “do this after that”. If the order of statements won’t cut it, what will?

Callbacks

Well, if the order of statements won’t cut it, maybe callbacks, events, Promises, ES6 Generators, or 3rd party libraries such as Async.js will? As you can see, there’s no shortage of tools to use. Let’s pick callbacks and see where that leads us.

A callback is a function used to effect control flow. You generally pass a callback function to another function which is expected to perform some asynchronous operation and call the callback with the result of the asynchronous operation. This effectively adds another frame on the stack which needs to be cleared before the async-op-performing function itself is removed from the stack.

If you take another look at inquirer.prompt above, you will see an example of callback usage. The second argument inquirer.prompt takes is a callback function which itself takes an argument which contains the data inquirer.prompt fetched from the user.

Callbacks are nice, until they aren’t. You might have heard the term “callback hell”, but in any case, here’s a quick look down the abyss:

Pretty isn’t it?

The point here isn’t that you shouldn’t use callbacks - it’s that you should be careful not to overuse them. Mixu puts it nicely in his online Node book:

When you start coding with Node.js, it’s a bit like learning programming the first time. Since you want everything to be asynchronous, you use a lot of callbacks without really thinking about how you should structure your code. It’s a bit like being overexcited about the if statement, and using it and only it to write complex programs.
Mixu’s Node book

OK, so with the background taken care of, let’s move on to one approach we can take to mitigate callback hell by using Lo-Dash/Underscore.js, something you’re likely to already be depending on.

The approach

To start off with, consider the snippet used in Adrian’s post:

// generate a callback
function cb(text) { 
    return function(next) { 
        console.warn(text); 
        next();
    }
}

// a list of things to call
var actions = [ cb(1), cb(2), cb(3), cb(4) ]

// call the functions in the series array in order
_(actions).reduceRight(_.wrap, function() { console.warn('done') })(); 

// prints out:
// ---
// 1
// 2
// 3
// 4
// done

The general idea is to store the functions you want to have execute sequentially in an array (actions) and use reduceRight and wrap to give you back a function which you can invoke to kick off the sequential execution of the functions in actions.

That’s the general idea. Let’s start breaking things down. (Note: I am quoting bits and pieces from the Lo-Dash docs below).

_.reduce

_.reduce(collection, [iteratee=_.identity], [accumulator], [thisArg])

This tells us that _.reduce can take 4 arguments, the collection to iterate over, the iteratee function, the accumulator which is the initial value the “reduction” starts with, and thisArg which is the this binding of iteratee.

The iteratee is bound to thisArg and invoked with four arguments; (accumulator, value, index|key, collection).

So apart from knowing what _.reduce takes, you’ll also need to know what the iteratee function takes.

If accumulator is not provided the first element of collection is used as the initial value.
This is referring to _.reduce‘s 3rd argument. If you don’t supply an argument, then the first element of collection will be used as the 1st argument in iteratee‘s first invocation.
iteratee‘s 2nd argument is value, i.e. it’s the next value in collection which hasn’t yet been through iteratee.
iteratee‘s 3rd argument is the index of value in the collection (“index” when the collection is an array, or “key” when the collection is an object).
iteratee‘s 4th argument is the collection itself (i.e. _.reduce‘s first argument).

Here’s a look at _.reduce in action:

var _ = require("lodash");

var nums = [1,2,3,4,5];

function sum(accumulator, value, index, collection) {
  console.log("acc", accumulator, "value", value, "index", index, "collection", collection);
  return accumulator + value;
}

var fromFirstEl = _.reduce(nums, sum);
console.log("fromFirstEl:", fromFirstEl);

var from5 = _.reduce(nums, sum, 5);
console.log("from5:", from5);

// acc 1 value 2 index 1 collection [ 1, 2, 3, 4, 5 ]
// acc 3 value 3 index 2 collection [ 1, 2, 3, 4, 5 ]
// acc 6 value 4 index 3 collection [ 1, 2, 3, 4, 5 ]
// acc 10 value 5 index 4 collection [ 1, 2, 3, 4, 5 ]
// fromFirstEl: 15
// acc 5 value 1 index 0 collection [ 1, 2, 3, 4, 5 ]
// acc 6 value 2 index 1 collection [ 1, 2, 3, 4, 5 ]
// acc 8 value 3 index 2 collection [ 1, 2, 3, 4, 5 ]
// acc 11 value 4 index 3 collection [ 1, 2, 3, 4, 5 ]
// acc 15 value 5 index 4 collection [ 1, 2, 3, 4, 5 ]
// from5: 20

Notice how the value returned from the iteratee function (i.e. sum) becomes the iteratee‘s accumulator the next time it is invoked on the next item in the collection.

Also notice how, in the case of no initial accumulator, it’s as if nums is one element short, and the element taken out of nums is used as the initial accumulator.

_.reduceRight is the exact operation but in reverse:

var rightFrom10 = _.reduceRight(nums, sum, 10);
console.log("rightFrom10:", rightFrom10);

// acc 10 value 5 index 4 collection [ 1, 2, 3, 4, 5 ]
// acc 15 value 4 index 3 collection [ 1, 2, 3, 4, 5 ]
// acc 19 value 3 index 2 collection [ 1, 2, 3, 4, 5 ]
// acc 22 value 2 index 1 collection [ 1, 2, 3, 4, 5 ]
// acc 24 value 1 index 0 collection [ 1, 2, 3, 4, 5 ]
// rightFrom10: 25

So basically, we now know that _.reduce is a function which takes a collection and reduces the values in that collection to a single value based on the logic in its given iteratee.

_.wrap

Let’s now turn our attention to wrap.

_.wrap(value, wrapper)

Creates a function that provides value to the wrapper function as its first argument. Any additional arguments provided to the function are appended to those provided to the wrapper function. The wrapper is invoked with the this binding of the created function.

An example is probably better (based off of Underscore.js’s doc as I felt it was easier than the one given in Lo-Dash’s):

var _ = require("lodash");

function hello(name, surname) {
  return "hello: " + name + (surname ? " " + surname : "");
}

var wrappedHello = _.wrap(hello, function(func) {
  return "before, " + func("moe") + ", after";
});

console.log(wrappedHello());

var wrappedHello2 = _.wrap(hello, function(func, name) {
  return "before, " + func(name) + ", after";
});

console.log(wrappedHello2("joe"));

var wrappedHello3 = _.wrap(hello, function(func, name, surname) {
  return "before, " + func(name, surname) + ", after";
});

console.log(wrappedHello3("joe", "doe"));

// before, hello: moe, after
// before, hello: joe, after
// before, hello: joe doe, after

Essentially, wrap takes the first argument given to it and passes it to it’s second argument which is expected to be the wrapper function (in which you can do any logic around value). Any arguments you pass to the created wrapper functions (e.g. wrappedHello2) get passed as arguments to the wrapper function used in _.wrap.

Putting it all together

Revisiting Adrian’s example:

function cb(text) { 
    return function(next) { 
        console.warn(text); 
        next();
    }
}

// a list of things to call
var actions = [ cb(1), cb(2), cb(3), cb(4) ]

// call the functions in the series array in order
_.reduceRight(actions, _.wrap, function() { console.warn('done') })(); 
// the above is equivalent to what's used in Adrian's example:
// _(actions).reduceRight(_.wrap, function() { console.warn('done') })();

Consider the first reduction:

// _.wrap is the iteratee
var reduced1 = _.wrap(
  // iteratee's accumulator
  function() {
    console.warn('done');
  },
  // iteratee's value
  function(next) {
    console.warn(4);
    next();
  }
);
// reduced1 = function() {
//   console.warn(4);
//   (function() {
//     console.warn('done');
//   })();
// }

The second reduction:

// _.wrap is the iteratee
var reduced2 = _.wrap(
  // iteratee's accumulator is reduced1
  function() {
    console.warn(4);
    (function() {
      console.warn('done');
    })();
  },
  // iteratee's next value
  function(next) {
    console.warn(3);
    next();
  }
);
// reduced2 = function() {
//   console.warn(3);
//   (function() {
//     console.warn(4);
//     (function() {
//       console.warn('done');
//     })();
//   })();
// }

etc… etc… until we finally get back a function that wraps all other functions in actions. Hopefully, you’re starting to see how this thing sticks together.

Final example

To drive the point home, let’s finish this with a less trivial, but completely useless, example:

var _ = require("lodash");
var inquirer = require("inquirer");
var fs = require("fs");
var path = require("path");

var questions = [
  {
    type: "input",
    name: "username",
    message: "Enter username"
  },
  {
    type: "password",
    name: "password",
    message: "Enter password"
  },
];

var actions = [
  function(next) {
    inquirer.prompt(questions, function(answers) {
      console.log("answers:", answers);
      next(answers);
    });
  },

  function(next, answers) {
    if(answers.password === "sdf") {

      fs.readFile(path.resolve(__dirname, __filename), "utf-8", function(err, data) {
        if(err) throw err;

        var re = new RegExp(answers.username);

        if(re.test(data)) {
          next("there was a match");
        } else {
          next("there was no match");
        }
      });

    }
  },

  function(next, matchRes) {
    console.log("matchRes", matchRes);
    next();
  }

];

_.reduceRight(actions, _.wrap, function() {
  console.log("done");
})();

You should now have a good idea of what’s going on. Note how next is always bound to the next function in actions which needs to execute after the function in question does. Any arguments after next (e.g. answers in the second function in actions) are passed in when next is invoked in a previous function’s body (i.e. next(answers) in actions‘s first function).

You can see that we are still using callbacks when using asynchronous functions like inquirer.prompt and fs.readFile, but we avoid nesting any deeper than the inital callback passed to these asynchronous functions.

Basically, this example asks for a username and password and proceeds to read it’s own source code if password is equal to "sdf". Also, assuming you get past the password check, an attempt to match the username against the contents of the file’s source code is carried out, and depending on the result, either "there was a match" or "there was no match" is passed to the next function in actions.

Since we’re never calling next if password does not equal "sdf", we’re effectively breaking the chain of wrapped functions.

Marionette CompositeView an example

2015-02-08T00:00:00.000Z

Intro

This post is based off of an example used in the book Better Backbone Applications with MarionetteJS by Joe Zimmerman. More specifically, it’s the recursive data example rendered by a combination of Collection and Composite views in Chapter 6 (itself based off of the following blog post by Derick Bailey: Composite Views: Tree Structures, Tables, And More).

I wasn’t sure I understood what was really going on in that example, so after reading the chapter, I decided to type it in and play around a bit. This post is just a description of that process (the end result is up on Github).

I will be implementing it using AMD modules and including the few changes necessary to get it to work in Marionette version 2.3.2 (the latest at time of writing). For a more detailed explanation of the example and the Marionette concepts used in it, have a look at Better Backbone Applications with MarionetteJS. I am in no way affilated with a third party and receive 0 benefit if you decide to purchase. That said, I feel confident recommending it based off of the value I’ve received from reading it.

High level aim

Basically, we want to go from:

[
    {
      nodeName: "1",
      nodes: [
        {
          nodeName: "1.1",
          nodes: [
            { nodeName: "1.1.1" },
            { nodeName: "1.1.2" },
            { nodeName: "1.1.3" }
          ]
        },
        {
          nodeName: "1.2",
          nodes: [
            { nodeName: "1.2.1" },
            {
              nodeName: "1.2.2",
              nodes: [
                { nodeName: "1.2.2.1" },
                { nodeName: "1.2.2.2" },
                { nodeName: "1.2.2.3" }
              ]
            },
            { nodeName: "1.2.3" }
          ]
        }
      ]
    },
    {
      nodeName: "2",
      nodes: [
        {
          nodeName: "2.1",
          nodes: [
            { nodeName: "2.1.1" },
            { nodeName: "2.1.2" },
            { nodeName: "2.1.3" }
          ]
        },
        {
          nodeName: "2.2",
          nodes: [
            { nodeName: "2.2.1" },
            { nodeName: "2.2.2" },
            { nodeName: "2.2.3" }
          ]
        }
      ]
    }
  ]

to:

Setting up

Install Bower globally if you don’t already have it: npm i -g bower
npm init
mkdir -p app/lib app/js
vim .bowerrc

1
2
3

{
  "directory": "app/lib"
}

bower init
bower i --save marionette lodash requirejs

1
2
3

compositeview-eg$ ls app/lib/
backbone            backbone.wreqr      lodash              requirejs
backbone.babysitter jquery              marionette          underscore

I am using both Npm and Bower to manage dependencies. Dependencies brought in by Npm will be used in developing the project (e.g. bringing in Grunt to automate tasks, or installing Bower so our project can use a different kind of dependency manager). Dependencies brought in by Bower will be used by the web application we’re building (i.e. these are the files consumed by the browser when it loads our application). If you’re thinking “why use 2 separate dependency managers?”, have a look at this SO question.

Basically, Npm makes use of nested dependencies which allows each dependency to have a copy of its own dependencies while Bower uses flat dependencies which means that when you bring in a dependency with Bower it will not include its own dependencies. So in the e.g. above, using Bower to bring in Marionette also gives us its dependencies, e.g. Underscore and Backbone, but these dependencies are outside of Marionette. Backbone, which also depends on Underscore, does not have its own copy of Underscore. So Underscore is shared by any dependency which needs it. This is great for pulling in dependencies which will be consumed by a browser since you do not want to load in several copies of a dependency in your application. On the server side, however, size doesn’t matter as much and using Npm allows us to cleanly use dependencies which have the correct version of their own dependencies (there are cases when this is not possible, e.g. plugins. You can read more about this at the NodeJS blog).

Next up, we’ll configure our AMD loader and log out one of our dependencies to get some kind of indication that we’re on the right track.

vim app/js/main.js

/*global require*/
"use strict";

require.config({
  paths: {
    jquery: "../lib/jquery/dist/jquery",
    backbone: "../lib/backbone/backbone",
    marionette: "../lib/marionette/lib/core/backbone.marionette",
    "backbone.wreqr": "../lib/backbone.wreqr/lib/backbone.wreqr",
    "backbone.babysitter": "../lib/backbone.babysitter/lib/backbone.babysitter",
    underscore: "../lib/lodash/lodash"
  }
});

require([
  "marionette"
], function (Marionette) {

  console.log("Marionette:", Marionette);

});

If you look inside app/lib/marionette you will notice that there are two Marionette scripts we could be pulling in, one in marionette/lib and one in marionette/lib/core. Both of these depend on Babysitter and Wreqr but the one in core pulls them in while the one outside core has these dependencies baked into its own source.

These pre-requisites are still required for Marionette to run, but this allows you to download them separately and update them independently.
MarionetteJS

Also, if you open up marionette/lib/core/backbone.marionette.js you will notice that it is requiring these pre-requisites using the following aliases:

1 2	var Wreqr = require('backbone.wreqr'); var BabySitter = require('backbone.babysitter');

… which is why we’re using the same alias names for the paths to these two dependencies in our AMD configuration.

The last thing we need in place to have something to serve is our index file: vim app/index.html


<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>title>
head>
<body>
  
  <script data-main="js/main" src="lib/requirejs/require.js">script>
  
body>
html>

The



Grunt watch those templates
2015-01-18T00:00:00.000Z
Purpose
Setting up automatic EJS (Underscore/Lo-Dash) and Handlebars template compilation in Grunt using grunt-contrib-watch.
The aim is to be able to spawn a process which will watch our templates and compile them to JST (Javascript template) files while we’re editing them. That way we can write our templates in a more readable fashion but use the actual JST files at runtime (thus avoiding the need to compile them dynamically).
Starting to watch
cd into an empty directory and:
1
2
3
$ npm init
$ npm i --save-dev grunt grunt-contrib-jst grunt-contrib-watch load-grunt-tasks grunt-contrib-handlebars
$ mkdir -p app/scripts
Start the Gruntfile.js configuration with the following, just to check that watch is working:
vim Gruntfile.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
'use strict';

module.exports = function (grunt) {

  require('load-grunt-tasks')(grunt);

  grunt.initConfig({
    watch: {
      options: {
        nospawn: true
      },
      log: {
        files: [
          'app/scripts/**/*.js'
        ],
        tasks: ['tmpLog']
      }
    }
  });

  grunt.registerTask('tmpLog', function () {
    grunt.log.write('watchin is workin');
  });

};
Running grunt watch and editing a JS file in ‘app/scripts/**/*.js’ (e.g. app/scripts/tmp.js) should give the following output:
Compiling EJS templates
Replace watch’s log option with the following:
1
2
3
4
5
6
jst: {
  files: [
    'app/scripts/templates/ejs/**/*.ejs'
  ],
  tasks: ['jst:compile']
}
Configure JST compilation for EJS files with the following (don’t include amd: true if you’re not working with an AMD module loader like RequireJS):
1
2
3
4
5
6
7
8
9
10
jst: {
  options: {
    amd: true
  },
  compile: {
    files: {
      'app/scripts/templates/jst/ejsTemplates.js': ['app/scripts/templates/ejs/**/*.ejs']
    }
  }
}
Personally, I find using jst for the key a bit confusing. The way I understand it is that both grunt-contrib-jst and grunt-contrib-handlebars, as well as any other JST compiler, compiles some kind of template file to a Javascript file in order to make the authoring and maintaing of these JST files easier (since these Javascript files tend to be heavy on string concatenation and such).
Since grunt-contrib-jst compiles EJS templates, it seems to me that it would have made more sense to call the plugin grunt-contrib-ejs and to use ejs as the key to configure this plugin in grunt.initConfig. But the key to use for the plugin is jst, so I’m also sticking to jst in the watch plugin configuration. I will, however, be suffixing these template files with .ejs, hence the above configuration.
Running grunt watch and editing an EJS file in our watched path should trigger the compilation, e.g:
vim app/scripts/templates/ejs/tmp.ejs
1
<ul>
    <% for(var i=0; i<supplies.length; i++) { %>
        <li>
            <a href='supplies/<%= supplies[i] %>'>
                <%= supplies[i] %>
            </a>
        </li>
    <% } %>
</ul>
Compiling Handlebars templates
Add the following option to the watch task configuration:
1
2
3
4
5
6
handlebars: {
  files: [
    'app/scripts/templates/hbs/**/*.hbs'
  ],
  tasks: ['handlebars:compile']
}
Then configure the grunt-contrib-handlebars plugin itself (again, I want the generated JST file to be wrapped in an AMD define, YMMV):
1
2
3
4
5
6
7
8
9
10
handlebars: {
  options: {
    amd: true
  },
  compile: {
    files: {
      'app/scripts/templates/jst/hbsTemplates.js': ['app/scripts/templates/hbs/**/*.hbs']
    }
  }
}
Try it out:
vim app/scripts/templates/hbs/tmp.hbs
1
2
3
4
5
6
<div class="entry">
  <h1>{{title}}h1>
  <div class="body">
    {{body}}
  div>
div>
… and assuming you’ve restarted grunt watch and it’s running in the background, you should get hbsTemplates.js.
Final Gruntfile.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
'use strict';

module.exports = function (grunt) {

  require('load-grunt-tasks')(grunt);

  grunt.initConfig({

    watch: {
      options: {
        nospawn: true
      },
      jst: {
        files: [
          'app/scripts/templates/ejs/**/*.ejs'
        ],
        tasks: ['jst:compile']
      },
      handlebars: {
        files: [
          'app/scripts/templates/hbs/**/*.hbs'
        ],
        tasks: ['handlebars:compile']
      }
    },

    jst: {
      options: {
        amd: true
      },
      compile: {
        files: {
          'app/scripts/templates/jst/ejsTemplates.js': ['app/scripts/templates/ejs/**/*.ejs']
        }
      }
    },

    handlebars: {
      options: {
        amd: true
      },
      compile: {
        files: {
          'app/scripts/templates/jst/hbsTemplates.js': ['app/scripts/templates/hbs/**/*.hbs']
        }
      }
    }

  });

};



Getting started with Assemble
2015-01-10T00:00:00.000Z
Version of Assemble at time of writing: 0.4.42
If you want to follow along with the end result at hand, the repo is up on Github.
Intro
Assemble, in their own words, is a static site generator for Grunt.js, Yeoman and Node.js. This blog post is a record of the approach taken and material covered during my first time usage of Assemble.
grunt-init-assemble
The first approach I took was to have a go at the grunt-init template for Assemble, grunt-init-assemble:
npm i -g grunt-cli grunt-init if you don’t already have them installed.
git clone https://github.com/ghost-town/grunt-init-assemble ~/.grunt-init/assembleAllows us to scaffold a new Assemble project using grunt-init.
cd into an empty directory and grunt-init assembleThis will look in ~/.grunt-init by default.
npm install && bower install to bring in Node and Bower packages.
grunt setup runs a task which is defined in the generated Gruntfile.js.
Go though your Gruntfile.js and delete the sections marked for deletion in the comments (they were there just for running grunt setup and should be removed after executing this task).
This leaves us with a bootstrapped Assemble project. If you’re not familiar with Assemble, it’s a good idea to have a look at the generated project. The rest of this post builds an example project from scratch i.e. without starting off using grunt-init-assemble, however, I am using the generated project as a guide - “cherry picking” if you will.
Starting from scratch
The idea is to run Assemble as part of a Grunt build and end up with a page assembled from bits and pieces with the help of Assemble. Along the way, we’ll be taking a look at what goes in to working with an Assemble project.
_config.yml and Gruntfile.js
npm init && npm i --save-dev grunt grunt-contrib-clean assemble
vim _config.yml1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root:             public
dest:             <%= site.root %>
assets:           <%= site.dest %>/assets

data:             data/*.{json,yml}

templates:        templates
includes:         <%= site.templates %>/includes/*.hbs
layouts:          <%= site.templates %>/layouts
layout:           default.hbs

helpers:          
  - <%= site.templates %>/helpers/*.js

description:      <%= pkg.description %>
Apart from pulling in some dependencies we’re setting up a _config.yml from which we’ll be accessing values to set up our Assemble configuration in the Gruntfile. Speaking of which, lets get a minimal Gruntfile in place. From the snippet below, notice how the site variable references the object built from parsing _config.yml. So when used in _config.yml above, you can think of site as being sort of like this, a reference to itself - again I am basing my approach off of the grunt-init-assemble example above.
vim Gruntfile.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
module.exports = function(grunt) {
'use strict';

  // Project configuration.
  grunt.initConfig({

    // Project metadata
    pkg   : grunt.file.readJSON('package.json'),
    site  : grunt.file.readYAML('_config.yml'),

    // Before generating any new files, remove files from previous build.
    clean: {
      example: ['<%= site.dest %>/*']
    },

    // Build output from templates and data
    assemble: {
      options: {
        flatten: true,
        production: false,
        assets: '<%= site.assets %>',

        // Metadata
        pkg: '<%= pkg %>',
        site: '<%= site %>',
        data: ['<%= site.data %>'],

        // Templates
        partials: '<%= site.includes %>',
        layoutdir: '<%= site.layouts %>',
        layout: '<%= site.layout %>',

        // Extensions
        helpers: '<%= site.helpers %>'
      },
      example: {
        files: {
          '<%= site.dest %>': ['<%= site.templates %>/example.hbs']
        }
      }
    }
  });

  grunt.loadNpmTasks('grunt-contrib-clean');
  grunt.loadNpmTasks('assemble');

  grunt.registerTask('default', ['clean', 'assemble']);

};
As you can see, we’re freely making use of the site and pkg variables in both _config.yml and Gruntfile.js, accessing them within Lo-Dash / Underscore templating.
Layouts and Partials
Next we’ll create the layoutdir and partials directories which will house our layouts and partials respectively. A layout is a full page with placeholders to fill in. Partials are re-usable parts of pages which can be included within other pages.
The Assemble documentation lists both layouts and partials, along with some other concepts, under the “Templates” heading. From this we can assume that all these parts of Assemble can be grouped under the term “Template”. Hence, our directory structure will reflect this (plus I’m just copying the grunt-init-assemble project again):
mkdir -p templates/layouts templates/includes
vim templates/layouts/default.hbs
 1
2
3
4
{{> preamble}}
### Begin
{{> body }}
### End
vim templates/includes/preamble.hbs
 1
2
3
{{filename}} is just an example.

---
As you might have guessed, {{> preamble}} and {{> body}} are placeholders which will get filled in with contents from other files, as we'll see later on. {{filename}} is a variable which will resolve to the name of the file being generated.The example.hbs page
Lets now fill in another missing piece from our Grunt configuration, the example.hbs file. Note the example target in the assemble multi task. In the example target we are specifying that we want a file to be generated from ['<%= site.templates %>/example.hbs'] i.e. from templates/example.hbs.  Of course, we could have added more input files in the array and/or made use of globbing patterns to match multiple files - but lets keep things simple.
echo "testing testing" > templates/example.hbs
gruntNote, you will need to install grunt-cli globally if you don’t already have it: npm i -g grunt-cli
Assuming the directory public does not already exist, the file public.html is created. If the public directory exists when you run the Assemble build, then public/example.html is created. I find this a bit confusing so I prefer to explicitly name the generated file in the assemble.example target like so:
1
2
3
4
5
6
// ...
      example: {
        files: {
          '<%= site.dest %>/example': ['<%= site.templates %>/example.hbs']
        }
      }
I exclude the file extension because it is not picked up from here. The ext option is used for this. assemble.options configures the global settings for the assemble target. These are settings which will apply to all targets in this task unless the task itself overwrites certain settings, like so:
1
2
3
4
5
6
7
8
9
// ...
      example: {
        options: {
          ext: '.md'
        },
        files: {
          '<%= site.dest %>/example': ['<%= site.templates %>/example.hbs']
        }
      }
Up until now, because we weren’t specifying the ‘ext’ option globally, the default ‘.html’ has always been applied. However, since we’re now overwritting this option at the target level, running grunt gives us public/example.md with the contents shown below: 
1
2
3
4
5
6
7
8
9
example.md is just an example.

---


### Begin
testing testing

### End
{{> preamble}} in default.hbs was substituted with the contents of the preamble.hbs partial and {{> body}} was substituted with the contents of example.hbs, the page that was rendered using the default.hbs layout. 
Also, there's a small example of using variables in the preamble partial i.e. {{filename}} which resolves to example.md, the name of the file being generated.Spicing up example.hbs
Finally, lets change example.hbs and make it slightly more interesting:
vim templates/example.hbs
1
2
3
4
5
6
7
8
9
10
11
---
title: Example
content: ['content/todo-today.md', 'content/todo-tomorrow.md']
---

### {{title}}

{{site.description}}
{{#each content}}
{{str this}}
{{/each}}
The file starts out by defining two variables, title and content, in the YAML front matter. Both these variables, as well as the site variable defined in our Gruntfile, are used in the page. You will want to make sure your package.json has a value for the description key if you want anything to show up for {{site.description}}. In my case, I have: 
1
"description": "This is a dummy project meant to explore Assemble."
The content variable is being used in a Handlebars block helper to iterate over its values. {{str}} is a custom helper which just reads the contents of the file whose path is in this and includes it in the page being built:
vim templates/helpers/str.js
1
2
3
4
5
6
7
8
9
10
var fs = require('fs');

module.exports.register = function (Handlebars, options, params) {
  'use strict';

  Handlebars.registerHelper('str', function(filePath) {
    var res = fs.readFileSync(filePath, 'utf8');
    return new Handlebars.SafeString(res);
  });
};
Most likely, there is already a helper which does this for us. I have not yet been through the helpers in the handlebars-helpers project. I figured it was just as well to do this “by hand” so to speak, to at least get some experience doing it and demoing it here.
After filling in our two content files, we can kick off the build process again and get some more interesting results:
vim content/todo-today.md
1
2
* Finish documenting Assemble findings in blog post
* Answer emails
vim content/todo-tomorrow.md
1
2
* Spend 30 mins on X
* Review this week's progress on Y
grunt && cat public/example.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
example.md is just an example.

---


### Begin


### Example

This is a dummy project meant to explore Assemble.

* Finish documenting Assemble findings in blog post
* Answer emails


* Spend 30 mins on X
* Review this week's progress on Y



### End
A bit more whitespace than I would have liked, but given how customizable all this seems to be, something can most likely be done about that (or maybe pass the output through some kind of postprocessing).
If you’ve used Assemble before and know how this can be improved (in general, but specifically the {{str}} and whitespace parts), please leave a comment below. I will update and give attribution. Thanks!


Setting up Karma with Mocha, Chai, and RequireJS
2014-12-26T00:00:00.000Z
Following along
If you want to follow along with the end result at hand, the repo is up on Github. You will need to install dependencies with npm install && bower install
Versions 
Karma 0.12.28
Node 0.10.33
NPM 1.4.28
Bower 1.3.12
Purpose
Going from zero to a project which supports running Mocha/Chai tests on a browser through Karma, with modularity concerns handled by RequireJS.
Other resources
http://karma-runner.github.io/0.12/plus/requirejs.html and accompanying repo https://github.com/kjbekkelund/karma-requirejsGreat resources to start off with. It’s using Jasmine though. I’d like to set this up with Mocha/Chai so it’s not exactly what I’m looking for. I still borrow heavily from these two resources here.
http://attackofzach.com/setting-up-a-project-using-karma-with-mocha-and-chai/This one is interesting and it works (I don’t remember if I had to change anything but if I did - nothing major). The only point where it fell short for me was that I needed to have RequireJS in the mix.
https://github.com/x2es/boilerplate-karma-mocha-chai-requirejs.git
This seemed really promising at first, but I’ve had no luck getting it to work. After npm install and karma start I get:
/Users/justin/tmp/boilerplate-karma-mocha-chai-requirejs/node_modules/karma/node_modules/di/lib/injector.js:9
throw error(‘No provider for “‘ + name + ‘“!’);
Then I figure out it’s missing some dependencies (maybe it was made to work against an older version of Karma even though it’s set to ‘latest’ in package.json). So I npm i --save-dev karma-requirejs karma-chai karma-chrome-launcher to see what happens when the missing dependencies are installed, but running Karma again gives me:
Chrome 39.0.2171 (Mac OS X 10.10.1) ERROR
Uncaught Error: Mismatched anonymous define() module: function () {
return (root.sinon = factory());
}
http://requirejs.org/docs/errors.html#mismatch
at /Users/justin/tmp/boilerplate-karma-mocha-chai-requirejs/node_modules/requirejs/require.js:141
ERROR [karma]: [TypeError: Cannot set property ‘results’ of undefined]
At this point I decided it would be better to try set this up from scratch. I’d still have to do this to better understand what’s going on.
Setting up Karma
Install karma-cli globally if you don’t already have it npm i -g karma-cli. This will allow us to use karma from our projects which have karma installed locally (i.e. instead of ./node_modules/karma/bin/karma). Then set up the project:
npm init
npm i --save-dev karma
karma init
Which testing framework do you want to use ?
Press tab to list possible options. Enter to move to the next question.
> mocha
Do you want to use Require.js ?
This will add Require.js plugin.
Press tab to list possible options. Enter to move to the next question.
> yes
Do you want to capture any browsers automatically ?
Press tab to list possible options. Enter empty string to move to the next question.
> Chrome
>
What is the location of your source and test files ?
You can use glob patterns, eg. “js/*.js” or “test/**/*Spec.js”.
Enter empty string to move to the next question.
> src/**/*.js
> test/**/*Spec.js
> lib/**/*.js
>
Should any of the files included by the previous patterns be excluded ?
 You can use glob patterns, eg. “**/*.swp”.
 Enter empty string to move to the next question.
src/main.js
>
Do you wanna generate a bootstrap file for RequireJS?
 This will generate test-main.js/coffee that configures RequireJS and starts the tests.
> yes
Do you want Karma to watch all the files and run the tests on change ?
 Press tab to list possible options.
> yes
 Ignore any warning messages related to patterns not matching any files (they won’t since we don’t have them yet).
This will bring in the Node dependencies we need as well as set us up with a karma.conf.js and test-main.js file.
karma.conf.js
This file contains our Karma configuration as per the karma init choices we made. Note the files key which lists the files which are loaded by the Karma server (some are included in the browser, some are not depending on the ‘included’ key shown in the snippet below). ‘test-main.js’ has the default included value of true while the files matched by the other three patterns are not included in the browser using script tags (they are watched for changes though and can be served from the Karma server (e.g. by requiring them via RequireJS)). You can read up more on this part of the config file here. I’d rather have ‘test-main.js’ in the ‘test’ directory though, so lets move it and mirror the change in the config:
mkdir test
mv test-main.js test/test-main.js
vim karma.conf.js
 1
2
3
4
5
6
7
files: [
  'test/test-main.js',
  {pattern: 'src/**/*.js', included: false},
  {pattern: 'test/**/*Spec.js', included: false},
  // include lib/**/*.js otherwise trying to load jquery etc.. from test's RequireJS will fail
  {pattern: 'lib/**/*.js', included: false}
],
Also note that we’re excluding ‘src/main.js’, the file that will contain our RequireJS configuration for when we’ll be bringing in our dependencies from our app’s HTML file. For the purposes of testing, that configuration will be done in ‘test/test-main.js’ instead.
test-main.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
var allTestFiles = [];
var TEST_REGEXP = /(spec|test)\.js$/i;

var pathToModule = function(path) {
  return path.replace(/^\/base\//, '').replace(/\.js$/, '');
};

Object.keys(window.__karma__.files).forEach(function(file) {
  if (TEST_REGEXP.test(file)) {
    // Normalize paths to RequireJS module names.
    allTestFiles.push(pathToModule(file));
  }
});

require.config({
  // Karma serves files under /base, which is the basePath from your config file
  // Look at your browser's debugging console if you have trouble loading in files via RequireJS
  // (in our case, we are starting Chrome from Karma).
  baseUrl: '/base',

  // dynamically load all test files
  deps: allTestFiles,

  // The tests are loaded in asynchronously via RequireJS 
  // so we need to indicate which function to run to kick things off once they have been loaded.
  // i.e. if you don't include this callback, the tests will not run
  // In our case, Mocha will be running the tests and the karma-mocha adapter has mapped the function to
  // kick things off to window.__karma__.start
  callback: window.__karma__.start
});
The dependencies (i.e. the tests we want to run) are evaluated dynamically from the files we specified for inclusion in our karma.conf.js file. More specifically, we are filtering out from these files those which match the TEST_REGEXP expression (i.e. any files which end with ‘spec.js’ or ‘test.js’ irrespective of this suffix’s case). The final part of this dependency evaluation is noramlizing the file path to be a correct RequireJS module name by taking off the ‘/base/‘ prefix and ‘.js’ suffix.
After these dependencies are loaded, we are executing the ‘window.__karma__.start’ function to run our tests as specified in the callback key.
Setting up front-end dependencies
Now that Karma’s in place, we’ll want to start pulling in our front-end dependencies. I’ll be using Bower to do that, changing the directory in which Bower installs dependencies to ‘lib’:
Install Bower globally if you don’t already have it (you’ll probably be using this in many projects so it’s best to have it globally installed anyway): npm install -g bower
bower init
vim .bowerrc
 1
2
3
{
    "directory": "lib"
}
bower i --save-dev jquery lodash requirejs
This will create the directory ‘lib’ if necessary
Note: the ‘requirejs’ we’re pulling in here is not the one which will be used in our Karma tests.
You should now have some front-end packages in your lib directory and the appropriate changes in bower.json.
Also, maybe this is worth addressing. As noted above, the RequireJS we’re pulling in for our front-end app is not the same RequireJS we’re running in our Karma test. When we’ll actually be taking Karma for a spin, you’ll be able to inspect your browser’s console and see that it’s loading the RequireJS in our node_modules. This is set up via the karma-requirejs plugin we’re referring to in our frameworks key in karma.conf.js.
So basically, the RequireJS we’re pulling in via Bower is the one we’ll be making use of from our index.html.
Dummy test
For the sake of having something to run, I will bring in some code from this repo: https://github.com/kjbekkelund/karma-requirejs
I’d like to work with Mocha/Chai though, so I’ll be making some changes.
Lets start with src/app.js (also shown below):
mkdir src
vim src/app.js
 1
2
3
4
5
6
7
8
9
10
11
define(function() {
    var App = function(el) {
        this.el = el;
    };

    App.prototype.render = function() {
        this.el.html('require.js up and running');
    };

    return App;
});
Next up, we’ll add test/appSpec.js, with some changes to use Chai’s syntax instead of Jasmine’s (and requiring Lo-Dash instead of Underscore):
vim src/appSpec.js
  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
define(['app', 'jquery', 'lodash'], function(App, $, _) {
  describe('just checking', function() {

    it('works for app', function() {
      var el = $('
');

      var app = new App(el);
      app.render();

      expect(el.text()).to.equal('require.js up and running');
    });

    it('works for lodash', function() {
      expect(_.size([1,2,3])).to.equal(3);
    });

  });
});
Note that we are accessing ‘describe’ and ‘expect’ globally since they’ve been bound to the ‘window’ in their respective Karma plugin files. Our modules and dependencies, though, are pulled in via RequireJS.
Bringing in Chai, a caveat, and the bacon
So now we’re just one step away from running our tests on Karma with this set-up. We still haven’t installed Chai and set it up in our Karma config, so go ahead and do that now:
npm i --save-dev karma-chai chai
vim karma.conf.js
 1
frameworks: ['mocha', 'requirejs', 'chai'],
Note: the order you list the plugins is important. Put ‘chai’ after ‘requirejs’. When I switch these around (as I had done originally), I get:
Uncaught TypeError: Cannot read property ‘should’ of undefined
  at /Users/justin/tmp/setting-up-karma/node_modules/karma-chai/adapter.js:4
I found this out thanks to: https://github.com/xdissent/karma-chai/issues/5
Anyway, although it took longer than I would have liked (missing the ‘lib/**/*.js’ loading in karma.conf.js tripped me up quite a bit), we can kick off Karma with karma start and leave it running while we develop our tests and have them executed in real, possibly multiple, browsers.
The repo for this example is up on Github - remember to get the dependencies: npm install && bower install. I have added the following two files to this repo which we haven’t talked about yet. Again they are based off of the ones in the karma-requirejs example but adapted a bit for this example:
vim index.html
1
2
3
4
5
6
7
8
9
10
11

<html>
  <head>
    <meta charset="utf-8">
    <title>Karma example setup with Require.jstitle>
    <script data-main="src/main.js" src="lib/requirejs/require.js">script>
  head>

  <body>
  body>
html>
vim src/main.js
1
2
3
4
5
6
7
8
9
10
11
12
requirejs.config({
    paths: {
        'jquery': '../lib/jquery/dist/jquery',
        'lodash': '../lib/lodash/dist/lodash'
    }
});

define(['app', 'jquery', 'lodash'], function (App, $, _) {
    var app = new App($('body'));
    app.render();
    console.log(_.size([1,2,3]));
});
If you open up index.html in a browser you should get the text “require.js up and running” and “3” in the console. This is using the RequireJS we pulled in via Bower.


To mock a Mongoose
2014-12-14T00:00:00.000Z
Purpose
Use Mongoose in a test without having to have MongoDB running. Mocking done with Mockgoose.
Setting up
Setting up can take a while sometimes so I decided to give a different approach a go this time round. I have created a simple grunt-init template which will do most of the grunt work for us (such pun!). Install grunt-init globally with npm i -g grunt-init if you don’t already have it. To use the template, simply clone it into your ~/.grunt-init/ directory (creating the directory if need be):
1
2
~$ mkdir .grunt-init && cd .grunt-init
~$ git clone https://github.com/justin-calleja/mongoose-blog-template-1
Automation time - simply cd into an empty directory and type the magic words:
1
2
to-mock-a-mongoose$ grunt-init mongoose-blog-template-1
to-mock-a-mongoose$ npm install
What have we got here?
First off, install mocha globally if you don’t already have it: npm i -g mocha and run mocha.
Open up model/db.js. This is where we’re establishing our db connection (and registering some event handlers). The connection will actually be mocked out in our test/test.js. That’s pretty useful. Now we don’t need MongoDB to be running to make some assertions on our code, code that would normally be interacting with the database.
To get the mocking done right, make sure you mock Mongoose before setting it up:
1
2
3
4
5
6
var mockgoose = require('mockgoose');
var mongoose = require('mongoose');

// mock mongoose before requiring the script which establishes the connection (to mock the connection)
mockgoose(mongoose);
require('../model/db');
model/item.js defines the structure (the schema) of an Item document which we’d like to store in our db. We’re also creating a model from our schema. This is basically a constructor we can use to create Item objects which will map to documents in our db. i.e. instances created from our Item model will be important to us if we care about interacting with our db.
test/test.js, after mocking our Mongoose, simply requires and makes use.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// code abbreviated... see generated test.js

// get a reference to our Item schema/model
var item = require('../model/item');

beforeEach(function(done) {
  mockgoose.reset();
  // create and insert two dummy docs
  item.model.create({ text: 'write blog on A' }, { text: 'write blog on B' }, function(err, blogOnA, blogOnB) {
    if(err) {
      console.log('Error creating documents in beforeEach: ' + error);
      throw(err);
    }
    done();
  });
});

describe('Blah', function() {
  describe('Bleh', function() {
    it('item.model.find() should give 2 documents back', function(done) {
      var query = item.model.find();
      query.exec(function(err, docs) {
        // expect to find the documents inserted in beforeEach
        expect(docs.length).to.equal(2);
        done();
      });
    });
  });
});



Taking the Express route out
2014-12-06T00:00:00.000Z
Purpose
Example of how to separate route code in Express.js using npm link.
Setting up
We can use the express generator to get up and running quickly with Express. Install it with npm install express-generator -g if you don’t already have it:
1
2
3
tmp$ express todo-server
tmp$ cd todo-server
todo-server$ npm install
It might be worth mentioning that at the time of writing, this installs express at version approx 4.9.0 or ~4.9.0.
At this point we could npm start to start our server. As defined in package.json, we’re basically running node ./bin/www.
We now want to start working on a separate module which will contain the logic behind the /items route in our application:
1
2
3
4
todo-server$ mkdir ../todo-items
todo-server$ cd ../todo-items
todo-items$ npm init
todo-items$ npm install --save express
Writing some code
1
todo-items$ vim index.js
1
2
3
4
5
6
7
8
9
10
var express = require('express');
var router = express.Router();

var items = require('./items.json');

router.get('/', function(req, res) {
  res.send(items);
});

module.exports = router;
1
todo-items$ vim items.json
1
2
3
4
{
    "task A": true,
    "task B": false
}
We’re just defining a GET on ‘/‘.  This means that, whichever route our todo-items module gets loaded on in todo-server, making a GET request on it will give us back our items.json data.
That’s great but todo-server doesn’t have todo-items installed so how’s it going to use it? We could publish todo-items on npm or host it in a Git repo on Github for example.
As you can see, we’re just experimenting with todo-items for now. Maybe later we’ll use a proper database. Maybe we’re not sure which database to go with. Point is, we don’t want to publish or host todo-items as it’s still early days, and besides, it would be better to avoid having to stay re-installing the module for every single change we make.
Enter npm link:
1
2
3
4
5
6
todo-items$ npm link
/Users/justin/.nvm/v0.10.33/lib/node_modules/todo-items -> /Users/justin/tmp/todo-items

todo-items$ cd ../todo-server
todo-server$ npm link todo-items
/Users/justin/tmp/todo-server/node_modules/todo-items -> /Users/justin/.nvm/v0.10.33/lib/node_modules/todo-items -> /Users/justin/tmp/todo-items
Executing it from todo-items we get a link to this directory from our global node_modules. npm link todo-items in todo-server gets us a link in todo-server‘s node_modules to the link in our global node_modules, effectively linking back to our todo-items directory containing our implementation for that module. 
End result - if you list the contents of todo-server‘s node_modules you’ll see we have our todo-items in there, albeit a link and not an actual directory structure as our other dependencies are installed as. Now, since the todo-items installation in our todo-server is just a link, any updates we make in todo-items will automatically take effect in our todo-server project.
Seeing it work
Simply require and use:
1
todo-server$ vim app.js
1
2
var items = require('todo-items');
app.use('/items', items);
Now we can npm start, hit http://localhost:3000/items in a browser, and get our items.json back.


Using Viper for some Go basics
2014-10-25T00:00:00.000Z
Intro
This post will walk you through the basic usage of Viper. It is intended for people who are new to the Go programming language (I am just starting out myself). This is because the purpose isn’t Viper per se, but the process of doing some of the basics in Go.
Assumptions
You have Go installed. In my case: go version go1.3.3 darwin/amd64
Setting up
First of all, we need to create a directory to house our work i.e. we need to create a workspace:
Go code must be kept inside a workspace. A workspace is a directory hierarchy with three directories at its root:
src
pkg
bin
How to Write Go Code
1
2
3
4
5
$ mkdir useviper_workspace

$ cd useviper_workspace

useviper_workspace$ mkdir src bin pkg
Next, lets create the directory which will actually house our source code for this post / your project. In doing so, it’s handy to keep the following best practice in mind:
The packages from the standard library are given short paths such as “fmt” and “net/http”. For your own packages, you must choose a base path that is unlikely to collide with future additions to the standard library or other external libraries.
If you keep your code in a source repository somewhere, then you should use the root of that source repository as your base path. For instance, if you have a GitHub account at github.com/user, that should be your base path.
Note that you don’t need to publish your code to a remote repository before you can build it. It’s just a good habit to organize your code as if you will publish it someday. In practice you can choose any arbitrary path name, as long as it is unique to the standard library and greater Go ecosystem.
How to Write Go Code
1
useviper_workspace$ mkdir -p src/github.com/justincalleja/useviper
I’ll go with github.com/justincalleja/useviper. Shortly, we will see that go can be used to fetch 3rd party source code which is publicly hosted and that the directory structure plays a role in this i.e. if you’re planning to push your code to some public repo, you’ll want to reflect that in the directory structure leading to the root of your project’s source code (e.g. github.com/username/projectname).
Before continuing, you’ll want to set your GOPATH environment variable:
The GOPATH environment variable specifies the location of your workspace.
Your workspace can be located wherever you like … Note that this must not be the same path as your Go installation.
How to Write Go Code
1
2
3
4
5
6
7
8
9
10
useviper_workspace$ echo $GOPATH
/Users/justin/go-stuff/useviper_workspace

useviper_workspace$ type set_gopath
set_gopath is a function
set_gopath ()
{
    export GOPATH=`pwd`;
    export PATH=$GOPATH/bin:$PATH
}
I already have my GOPATH set using the set_gopath bash function which is shown above. You can put this in any file which is sourced when you start your terminal (e.g. ~/.bashrc, ~/.profile etc…).
Running some code
Finally, we can get to writing some code. If you want to, initialize a git repo at src/github.com/< your-username >/useviper and go ahead and add useviper.go in there:
1
2
3
4
5
6
7
8
9
10
11
package main

import (
"fmt"
"github.com/spf13/viper"
)

func main() {
fmt.Println("in main")
viper.SetConfigName("config") // name of config file (without extension)
}
We’re just printing to standard out and making basic usage of Viper as per usage instructions (note that if you don’t use something you import the compiler will complain about it and won’t let you build).
Now “fmt” is built-in, but what about “github.com/spf13/viper”? How do we get that dependency?
1
2
useviper_workspace$ go get
can't load package: package .: no buildable Go source files in /Users/justin/go-stuff/useviper_workspace
If I try running go get from the root of my workspace I get the error shown above. It seems that go get is expecting Go lang source files to be in the directory you’re running it in. Try running it again in the useviper directory where we have our useviper.go file. This time it hangs for a while. That’s because go get is scanning the source files it finds and downloading, building and installing our dependencies and code into our workspace. If we print out the directory structure down to 3 levels we get:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
useviper_workspace$ tree -L 3
.
├── bin
│   └── useviper
├── pkg
│   └── darwin_amd64
│       ├── github.com
│       └── gopkg.in
└── src
    ├── github.com
    │   ├── BurntSushi
    │   ├── justincalleja
    │   ├── kr
    │   ├── mitchellh
    │   └── spf13
    └── gopkg.in
        └── yaml.v1
In our case, we didn’t really need go get to stay parsing and figuring out what dependencies our project needs as we have only one (which in turn has it’s own dependencies). We could have achieved the same result by running go get github.com/spf13/viper from the root of our workspace.
As you can see, our src, pkg, and bin directories have been populated. src now includes the source code of our dependencies recursively (Viper is in github.com/spf13/viper). pkg contains library binaries. These binaries cannot be executed but they can be linked to. bin is where executable binaries go. The main() function in useviper.go makes the result of building the file an executable binary, so it’s placed in bin and we can run it.
Since we’ve added $GOPATH/bin to our PATH environment variable (refer to the set_gopath executable function above), running useviper at the terminal is simply a matter of entering the file’s name: useviper, which gives us back “in main”.
Next, we want to be able to modify our source code and rebuild. Change your main() function to look like the following:
1
2
3
4
5
6
7
8
9
10
11
12
func main() {
viper.SetConfigName("config") // name of config file (without extension)
viper.AddConfigPath(".")      // path to look for the config file in

err := viper.ReadInConfig()
if err != nil {
fmt.Println("Config not found...")
} else {
name := viper.GetString("name")
fmt.Println("Config found, name = ", name)
}
}
In order to install (build and put in bin), we find ourselves in the same sort of situation as we did with go get. We can either rely on being in the directory our source files are in and run go install, or we can specify what to install explicitly. I like having the goinstall.sh script around in my workspace root (shown below).
1
2
3
4
5
6
7
#!/bin/bash
if [ $# -eq 0 ]
then
    echo "go install github.com/justincalleja/useviper" && go install github.com/justincalleja/useviper
else
    echo "go install github.com/justincalleja/$1" && go install github.com/justincalleja/$1
fi
I thought it might be useful to install with just the project name (and a default) from the project root… but I have yet to develop enough in Go to see what is actually helpful or not in the process of writing code.
Basically, all the following will install useviper in bin for us:
1
2
3
4
5
useviper_workspace$ goinstall.sh

useviper_workspace$ goinstall.sh useviper

useviper$ go install
If you useviper now you should get: “Config not found…”. But if you create a file named config.yaml in the directory in which you’re going to run useviper with the following contents:
1
name: viper
you should get back “Config found, name =  viper”. Note that we did not need to set the config file’s extension in our main() as Viper picks up YAML, TOML, or JSON files automatically.
That wraps up this quick introduction to some Go lang basics :)